[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:50:10 dist-test-slave-kk5c krb5kdc[782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:50:10 dist-test-slave-kk5c krb5kdc[782](info): set up 2 sockets
Jun 20 21:50:10 dist-test-slave-kk5c krb5kdc[782](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 21:50:11.956887 775 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:50:12 dist-test-slave-kk5c krb5kdc[782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:50:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:50:12.091609 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:35545
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:35545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:12.209827 795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:12.210183 795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:12.210260 795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:12.214555 795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:12.214674 795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:12.214712 795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:12.214758 795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:12.214794 795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:12.220588 795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:35545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:35545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:12.221823 795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:12.223153 795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:12.232753 803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:12.232714 800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:12.232700 801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:12.233211 795 server_base.cc:1034] running on GCE node
I20260620 21:50:12.233839 795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:12.235064 795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:12.236270 795 hybrid_clock.cc:648] HybridClock initialized: now 1781992212236225 us; error 60 us; skew 500 ppm
Jun 20 21:50:12 dist-test-slave-kk5c krb5kdc[782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:12.239341 795 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:50:12.240331 795 webserver.cc:460] Webserver started at http://127.0.193.254:39509/ using document root <none> and password file <none>
I20260620 21:50:12.240888 795 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:12.240949 795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:12.241124 795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:12.243299 795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "0e7a27229008404384deb3a8f8b067a9"
format_stamp: "Formatted at 2026-06-20 21:50:12 on dist-test-slave-kk5c"
server_key: "b3b073aae59716f97ad95c4aec1f8c7c"
server_key_iv: "4367e5b3c269734f69f31d2f649bb91d"
server_key_version: "encryptionkey@0"
I20260620 21:50:12.243854 795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "0e7a27229008404384deb3a8f8b067a9"
format_stamp: "Formatted at 2026-06-20 21:50:12 on dist-test-slave-kk5c"
server_key: "b3b073aae59716f97ad95c4aec1f8c7c"
server_key_iv: "4367e5b3c269734f69f31d2f649bb91d"
server_key_version: "encryptionkey@0"
I20260620 21:50:12.247958 795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 21:50:12.250541 810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:12.252063 795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:12.252205 795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "0e7a27229008404384deb3a8f8b067a9"
format_stamp: "Formatted at 2026-06-20 21:50:12 on dist-test-slave-kk5c"
server_key: "b3b073aae59716f97ad95c4aec1f8c7c"
server_key_iv: "4367e5b3c269734f69f31d2f649bb91d"
server_key_version: "encryptionkey@0"
I20260620 21:50:12.252315 795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:12.271790 795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:12.275357 795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:12.275540 795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:12.283592 795 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:35545
I20260620 21:50:12.283686 862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:35545 every 8 connection(s)
I20260620 21:50:12.284710 795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:12.287741 863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:12.289191 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 795
I20260620 21:50:12.289363 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:12.291009 775 external_mini_cluster.cc:1428] Setting key 999a5980cfbd3cd350f37660c635a656
I20260620 21:50:12.295317 863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9: Bootstrap starting.
I20260620 21:50:12.297755 863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:12.298487 863 log.cc:824] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:12.300932 863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9: No bootstrap required, opened a new log
I20260620 21:50:12.305081 863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7a27229008404384deb3a8f8b067a9" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35545 } }
I20260620 21:50:12.305383 863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:12.305433 863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7a27229008404384deb3a8f8b067a9, State: Initialized, Role: FOLLOWER
I20260620 21:50:12.305898 863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7a27229008404384deb3a8f8b067a9" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35545 } }
I20260620 21:50:12.306090 863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:12.306177 863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:12.306308 863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:12.307377 863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7a27229008404384deb3a8f8b067a9" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35545 } }
I20260620 21:50:12.307775 863 leader_election.cc:302] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e7a27229008404384deb3a8f8b067a9; no voters:
I20260620 21:50:12.308151 863 leader_election.cc:288] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:12.308365 868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:12.308621 868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [term 1 LEADER]: Becoming Leader. State: Replica: 0e7a27229008404384deb3a8f8b067a9, State: Running, Role: LEADER
I20260620 21:50:12.309077 868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7a27229008404384deb3a8f8b067a9" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35545 } }
I20260620 21:50:12.309607 863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:12.310916 869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e7a27229008404384deb3a8f8b067a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7a27229008404384deb3a8f8b067a9" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35545 } } }
I20260620 21:50:12.311053 869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:12.311733 870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e7a27229008404384deb3a8f8b067a9. Latest consensus state: current_term: 1 leader_uuid: "0e7a27229008404384deb3a8f8b067a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7a27229008404384deb3a8f8b067a9" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35545 } } }
I20260620 21:50:12.311928 870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:12.312446 878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 21:50:12 dist-test-slave-kk5c krb5kdc[782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:12.315953 878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:12.320660 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:12.295027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42542 (local address 127.0.193.254:35545)
0620 21:50:12.295692 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:12.295712 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:12.296681 (+ 969us) server_negotiation.cc:407] Connection header received
0620 21:50:12.300321 (+ 3640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:12.300344 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:12.300753 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:12.301127 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:12.303950 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:12.305459 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:12.307224 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:12.307932 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:12.308123 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:12.314148 (+ 6025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:12.314157 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:12.314177 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:12.314219 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:12.316657 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:12.317442 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:12.317450 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:12.317456 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:12.317623 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:12.318087 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:12.318090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:12.318091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:12.318493 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:12.318754 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:12.319378 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 21:50:12.319673 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":231,"threads_started":1}
I20260620 21:50:12.323000 878 catalog_manager.cc:1347] Generated new cluster ID: d8dabc6c62cd46a3bc3d76c07bdaf967
I20260620 21:50:12.323122 878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:12.339282 878 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:12.340785 878 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:12.345341 878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e7a27229008404384deb3a8f8b067a9: Generated new TSK 0
I20260620 21:50:12.346350 878 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:50:12.384377 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:35545
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:12.515407 889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:12.515771 889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:12.515862 889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:12.522238 889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:12.522368 889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:12.522475 889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:12.529790 889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:35545
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:12.531726 889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:12.532894 889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:12.545298 894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:12.545400 897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:12.545296 895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:12.545662 889 server_base.cc:1034] running on GCE node
I20260620 21:50:12.546099 889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:12.546837 889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:12.548026 889 hybrid_clock.cc:648] HybridClock initialized: now 1781992212548009 us; error 32 us; skew 500 ppm
Jun 20 21:50:12 dist-test-slave-kk5c krb5kdc[782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:12.551491 889 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:50:12.552511 889 webserver.cc:460] Webserver started at http://127.0.193.193:40869/ using document root <none> and password file <none>
I20260620 21:50:12.553143 889 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:12.553238 889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:12.553478 889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:12.555751 889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "cfe042b7f955468387ce76daf413c719"
format_stamp: "Formatted at 2026-06-20 21:50:12 on dist-test-slave-kk5c"
server_key: "f2838d318ca128958d9634b6e31efcae"
server_key_iv: "0b76318faf83165dd43ab7588f079218"
server_key_version: "encryptionkey@0"
I20260620 21:50:12.556453 889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "cfe042b7f955468387ce76daf413c719"
format_stamp: "Formatted at 2026-06-20 21:50:12 on dist-test-slave-kk5c"
server_key: "f2838d318ca128958d9634b6e31efcae"
server_key_iv: "0b76318faf83165dd43ab7588f079218"
server_key_version: "encryptionkey@0"
I20260620 21:50:12.561004 889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:50:12.565014 904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:12.567766 889 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 21:50:12.567955 889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "cfe042b7f955468387ce76daf413c719"
format_stamp: "Formatted at 2026-06-20 21:50:12 on dist-test-slave-kk5c"
server_key: "f2838d318ca128958d9634b6e31efcae"
server_key_iv: "0b76318faf83165dd43ab7588f079218"
server_key_version: "encryptionkey@0"
I20260620 21:50:12.568101 889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:12.588914 889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:12.592902 889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:12.593129 889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:12.593771 889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:12.594724 889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:12.594803 889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:12.594892 889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:12.594941 889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:12.606954 1017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:33877 every 8 connection(s)
I20260620 21:50:12.607079 889 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:33877
I20260620 21:50:12.608479 889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:12.613077 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 889
I20260620 21:50:12.613392 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:50:12.613758 775 external_mini_cluster.cc:1428] Setting key d8a9a71ba68b02bfa7bc1e9cc934d684
Jun 20 21:50:12 dist-test-slave-kk5c krb5kdc[782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
I20260620 21:50:12.629387 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:12.611888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:37057 (local address 127.0.193.254:35545)
0620 21:50:12.612072 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:12.612075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:12.613592 (+ 1517us) server_negotiation.cc:407] Connection header received
0620 21:50:12.615325 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:12.615328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:12.615401 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:12.615500 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:12.617364 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:12.617921 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:12.620044 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:12.620313 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:12.620661 (+ 348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:12.623878 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:12.623881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:12.623883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:12.623910 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:12.625878 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:12.626724 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:12.626726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:12.626728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:12.626816 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:12.627445 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:12.627447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:12.627448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:12.627656 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:12.627778 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:12.629000 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0620 21:50:12.629200 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 21:50:12.630285 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:12.612339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35545 (local address 127.0.193.193:37057)
0620 21:50:12.613366 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:12.613443 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:12.615029 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:12.615760 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:12.615771 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:12.616364 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:12.617151 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:12.617176 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:12.618074 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:12.618079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:12.619862 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:12.619878 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:12.620712 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:12.620717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:12.620927 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:12.621759 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:12.621780 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:12.623631 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:12.626038 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:12.626046 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:12.626072 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:12.626541 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:12.626938 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:12.626942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:12.626944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:12.627322 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:12.627778 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:12.627787 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:12.628526 (+ 739us) client_negotiation.cc:769] Sending connection context
0620 21:50:12.628941 (+ 415us) client_negotiation.cc:240] Negotiation successful
0620 21:50:12.629342 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":178,"threads_started":1}
I20260620 21:50:12.631872 1018 heartbeater.cc:347] Connected to a master server at 127.0.193.254:35545
I20260620 21:50:12.632167 1018 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:12.633064 1018 heartbeater.cc:511] Master 127.0.193.254:35545 requested a full tablet report, sending...
I20260620 21:50:12.635134 827 ts_manager.cc:194] Registered new tserver with Master: cfe042b7f955468387ce76daf413c719 (127.0.193.193:33877)
I20260620 21:50:12.636948 827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:37057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:50:12.647573 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:35545
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:12.768513 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:12.768826 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:12.768873 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:12.773255 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:12.773332 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:12.773396 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:12.779086 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:35545
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:12.780210 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:12.781239 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:12.792233 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:12.792250 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:12.792327 1023 server_base.cc:1034] running on GCE node
W20260620 21:50:12.792243 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:12.792913 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:12.793706 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:12.794875 1023 hybrid_clock.cc:648] HybridClock initialized: now 1781992212794834 us; error 53 us; skew 500 ppm
Jun 20 21:50:12 dist-test-slave-kk5c krb5kdc[782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:12.797842 1023 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:50:12.798874 1023 webserver.cc:460] Webserver started at http://127.0.193.194:41317/ using document root <none> and password file <none>
I20260620 21:50:12.799569 1023 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:12.799666 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:12.799918 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:12.802079 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "b874a94ae7514cc7970c4663156658e7"
format_stamp: "Formatted at 2026-06-20 21:50:12 on dist-test-slave-kk5c"
server_key: "0a2a6d57b1c3ce077102e9fa836787af"
server_key_iv: "f2454d2c7d08fd7325c7bb56afab7f53"
server_key_version: "encryptionkey@0"
I20260620 21:50:12.802685 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "b874a94ae7514cc7970c4663156658e7"
format_stamp: "Formatted at 2026-06-20 21:50:12 on dist-test-slave-kk5c"
server_key: "0a2a6d57b1c3ce077102e9fa836787af"
server_key_iv: "f2454d2c7d08fd7325c7bb56afab7f53"
server_key_version: "encryptionkey@0"
I20260620 21:50:12.806857 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 21:50:12.809666 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:12.811383 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 21:50:12.811533 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "b874a94ae7514cc7970c4663156658e7"
format_stamp: "Formatted at 2026-06-20 21:50:12 on dist-test-slave-kk5c"
server_key: "0a2a6d57b1c3ce077102e9fa836787af"
server_key_iv: "f2454d2c7d08fd7325c7bb56afab7f53"
server_key_version: "encryptionkey@0"
I20260620 21:50:12.811661 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:12.858614 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:12.862581 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:12.862757 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:12.863564 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:12.864477 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:12.864542 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:12.864590 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:12.864609 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:12.875569 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:35827
I20260620 21:50:12.875670 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:35827 every 8 connection(s)
I20260620 21:50:12.876963 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:12.886442 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 1023
I20260620 21:50:12.886546 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:50:12.886852 775 external_mini_cluster.cc:1428] Setting key 2000477d9be9e42d5b28c3d0a94dad85
Jun 20 21:50:12 dist-test-slave-kk5c krb5kdc[782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:12.893731 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:12.879140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:39135 (local address 127.0.193.254:35545)
0620 21:50:12.879334 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:12.879337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:12.880371 (+ 1034us) server_negotiation.cc:407] Connection header received
0620 21:50:12.882241 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:12.882245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:12.882315 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:12.882395 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:12.883896 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:12.884441 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:12.885781 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:12.886016 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:12.886105 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:12.889299 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:12.889303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:12.889305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:12.889331 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:12.891111 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:12.891890 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:12.891893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:12.891895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:12.891986 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:12.892419 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:12.892422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:12.892423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:12.892622 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:12.892740 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:12.893469 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 21:50:12.893577 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 21:50:12.895143 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:12.879622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35545 (local address 127.0.193.194:39135)
0620 21:50:12.880166 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:12.880220 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:12.881953 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:12.882590 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:12.882599 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:12.883027 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:12.883694 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:12.883724 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:12.884589 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:12.884592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:12.885638 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:12.885650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:12.886215 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:12.886218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:12.886386 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:12.887155 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:12.887173 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:12.889090 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:12.891319 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:12.891325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:12.891340 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:12.891743 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:12.892086 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:12.892090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:12.892092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:12.892303 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:12.892767 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:12.892777 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:12.893172 (+ 395us) client_negotiation.cc:769] Sending connection context
0620 21:50:12.893415 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 21:50:12.893830 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":142,"threads_started":1}
I20260620 21:50:12.896672 1152 heartbeater.cc:347] Connected to a master server at 127.0.193.254:35545
I20260620 21:50:12.896962 1152 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
I20260620 21:50:12.897585 1152 heartbeater.cc:511] Master 127.0.193.254:35545 requested a full tablet report, sending...
I20260620 21:50:12.898819 827 ts_manager.cc:194] Registered new tserver with Master: b874a94ae7514cc7970c4663156658e7 (127.0.193.194:35827)
I20260620 21:50:12.899564 827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:39135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:50:12.919701 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:35545
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:13.038069 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:13.038383 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:13.038432 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:13.042812 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:13.042891 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:13.042959 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:13.048696 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:35545
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:13.049865 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:13.050951 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:13.061981 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:13.062007 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:13.062101 1157 server_base.cc:1034] running on GCE node
W20260620 21:50:13.062152 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:13.062928 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:13.063757 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:13.064988 1157 hybrid_clock.cc:648] HybridClock initialized: now 1781992213064967 us; error 36 us; skew 500 ppm
Jun 20 21:50:13 dist-test-slave-kk5c krb5kdc[782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:13.068250 1157 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:50:13.069206 1157 webserver.cc:460] Webserver started at http://127.0.193.195:45403/ using document root <none> and password file <none>
I20260620 21:50:13.069826 1157 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:13.069892 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:13.070080 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:13.072290 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "819c2e220a424c54becae79f038904fd"
format_stamp: "Formatted at 2026-06-20 21:50:13 on dist-test-slave-kk5c"
server_key: "dcfdca6c94cd8d00b22b0775dfd57435"
server_key_iv: "e6c4d8d5110e785ec181d79ed941e68b"
server_key_version: "encryptionkey@0"
I20260620 21:50:13.072914 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "819c2e220a424c54becae79f038904fd"
format_stamp: "Formatted at 2026-06-20 21:50:13 on dist-test-slave-kk5c"
server_key: "dcfdca6c94cd8d00b22b0775dfd57435"
server_key_iv: "e6c4d8d5110e785ec181d79ed941e68b"
server_key_version: "encryptionkey@0"
I20260620 21:50:13.077824 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 21:50:13.080588 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:13.082175 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:50:13.082298 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "819c2e220a424c54becae79f038904fd"
format_stamp: "Formatted at 2026-06-20 21:50:13 on dist-test-slave-kk5c"
server_key: "dcfdca6c94cd8d00b22b0775dfd57435"
server_key_iv: "e6c4d8d5110e785ec181d79ed941e68b"
server_key_version: "encryptionkey@0"
I20260620 21:50:13.082468 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:13.114319 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:13.117975 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:13.118156 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:13.118692 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:13.119581 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:13.119632 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:13.119673 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:13.119688 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:13.130441 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:39469
I20260620 21:50:13.130483 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:39469 every 8 connection(s)
I20260620 21:50:13.131614 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:50:13.137240 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 1157
I20260620 21:50:13.137389 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:50:13.137741 775 external_mini_cluster.cc:1428] Setting key f6d7e046bee7a72a98012d5ff5ff5e1f
Jun 20 21:50:13 dist-test-slave-kk5c krb5kdc[782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:13.146934 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:13.133697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:39629 (local address 127.0.193.254:35545)
0620 21:50:13.133852 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:13.133856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:13.134819 (+ 963us) server_negotiation.cc:407] Connection header received
0620 21:50:13.136142 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:13.136145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:13.136207 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:13.136291 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:13.137663 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:13.138390 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:13.139986 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:13.140244 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:13.140365 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:13.143093 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:13.143097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:13.143099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:13.143126 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:13.144669 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:13.145271 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:13.145274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:13.145275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:13.145358 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:13.145739 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:13.145751 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:13.145752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:13.145954 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:13.146042 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:13.146681 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 21:50:13.146807 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:50:13.147866 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:13.134163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35545 (local address 127.0.193.195:39629)
0620 21:50:13.134651 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:13.134705 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:13.135946 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:13.136455 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:13.136461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:13.136851 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:13.137463 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:13.137486 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:13.138578 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:13.138581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:13.139853 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:13.139863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:13.140394 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:13.140397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:13.140633 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:13.141347 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:13.141365 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:13.142889 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:13.144800 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:13.144806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:13.144821 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:13.145160 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:13.145467 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:13.145469 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:13.145471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:13.145625 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:13.146061 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:13.146067 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:13.146369 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 21:50:13.146556 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 21:50:13.146865 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260620 21:50:13.149219 1286 heartbeater.cc:347] Connected to a master server at 127.0.193.254:35545
I20260620 21:50:13.149535 1286 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:13.150197 1286 heartbeater.cc:511] Master 127.0.193.254:35545 requested a full tablet report, sending...
I20260620 21:50:13.151492 827 ts_manager.cc:194] Registered new tserver with Master: 819c2e220a424c54becae79f038904fd (127.0.193.195:39469)
I20260620 21:50:13.152218 827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:39629
I20260620 21:50:13.152915 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:50:13 dist-test-slave-kk5c krb5kdc[782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992213, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:50:13 dist-test-slave-kk5c krb5kdc[782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992213, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.193.193@KRBTEST.COM
I20260620 21:50:13.181571 1293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:13.170284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.0.193.193:33877)
0620 21:50:13.170626 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:13.170632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:13.170657 (+ 25us) server_negotiation.cc:407] Connection header received
0620 21:50:13.170802 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:13.170807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:13.171006 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:13.171134 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:13.171998 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:13.172810 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:13.173859 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:13.174249 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:13.174354 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:13.177456 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:13.177468 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:13.177482 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:13.177569 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:13.179425 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:13.179949 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:13.179956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:13.179957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:13.180060 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:13.180462 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:13.180465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:13.180466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:13.180802 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:13.180993 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:13.181274 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 21:50:13.181415 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":123,"threads_started":1}
W20260620 21:50:13.182657 932 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48250
Jun 20 21:50:13 dist-test-slave-kk5c krb5kdc[782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:50:13 dist-test-slave-kk5c krb5kdc[782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.193@KRBTEST.COM
I20260620 21:50:13.211082 1293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:13.201358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.0.193.193:33877)
0620 21:50:13.201493 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:13.201507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:13.201676 (+ 169us) server_negotiation.cc:407] Connection header received
0620 21:50:13.201759 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:13.201762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:13.201810 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:13.201892 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:13.202754 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:13.203350 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:13.204245 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:13.204479 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:13.204636 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:13.207303 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:13.207310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:13.207313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:13.207348 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:13.209153 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:13.209691 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:13.209695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:13.209697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:13.209788 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:13.210093 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:13.210096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:13.210097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:13.210328 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:13.210518 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:13.210848 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 21:50:13.210956 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:50:13.212221 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 889
I20260620 21:50:13.218793 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 1023
I20260620 21:50:13.225629 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 1157
I20260620 21:50:13.231434 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 795
2026-06-20T21:50:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2388 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:50:13 dist-test-slave-kk5c krb5kdc[1300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:50:13 dist-test-slave-kk5c krb5kdc[1300](info): set up 2 sockets
Jun 20 21:50:13 dist-test-slave-kk5c krb5kdc[1300](info): commencing operation
krb5kdc: starting...
W20260620 21:50:14.354928 775 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:50:14 dist-test-slave-kk5c krb5kdc[1300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:50:14.492496 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:38729
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:38729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:14.623747 1313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:14.624097 1313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:14.624143 1313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:14.629299 1313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:14.629391 1313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:14.629413 1313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:14.629431 1313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:14.629447 1313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:14.636133 1313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:38729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:38729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:14.637359 1313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:14.638643 1313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:14.647967 1318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:14.647992 1319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:14.647996 1313 server_base.cc:1034] running on GCE node
W20260620 21:50:14.648015 1321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:14.648929 1313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:14.650152 1313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:14.651340 1313 hybrid_clock.cc:648] HybridClock initialized: now 1781992214651324 us; error 30 us; skew 500 ppm
Jun 20 21:50:14 dist-test-slave-kk5c krb5kdc[1300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:14.654675 1313 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:50:14.655977 1313 webserver.cc:460] Webserver started at http://127.0.193.254:46611/ using document root <none> and password file <none>
I20260620 21:50:14.656646 1313 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:14.656800 1313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:14.657065 1313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:14.659489 1313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "f322f137f4a44cccaed3c52684a64d0c"
format_stamp: "Formatted at 2026-06-20 21:50:14 on dist-test-slave-kk5c"
server_key: "906d4a8fcda291e6b86e2509dfcf6ea8"
server_key_iv: "9df09b818cb52e0f4e5b036e66c92fef"
server_key_version: "encryptionkey@0"
I20260620 21:50:14.660162 1313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "f322f137f4a44cccaed3c52684a64d0c"
format_stamp: "Formatted at 2026-06-20 21:50:14 on dist-test-slave-kk5c"
server_key: "906d4a8fcda291e6b86e2509dfcf6ea8"
server_key_iv: "9df09b818cb52e0f4e5b036e66c92fef"
server_key_version: "encryptionkey@0"
I20260620 21:50:14.664569 1313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 21:50:14.667650 1328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:14.668980 1313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 21:50:14.669179 1313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "f322f137f4a44cccaed3c52684a64d0c"
format_stamp: "Formatted at 2026-06-20 21:50:14 on dist-test-slave-kk5c"
server_key: "906d4a8fcda291e6b86e2509dfcf6ea8"
server_key_iv: "9df09b818cb52e0f4e5b036e66c92fef"
server_key_version: "encryptionkey@0"
I20260620 21:50:14.669323 1313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:14.682498 1313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:14.686203 1313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:14.686442 1313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:14.693965 1313 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:38729
I20260620 21:50:14.693977 1380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:38729 every 8 connection(s)
I20260620 21:50:14.695050 1313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:14.698179 1381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:14.700992 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 1313
I20260620 21:50:14.701170 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:14.701418 775 external_mini_cluster.cc:1428] Setting key ba4760a5e788bbcc92440f23f5e54482
I20260620 21:50:14.704504 1381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c: Bootstrap starting.
I20260620 21:50:14.706910 1381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:14.707753 1381 log.cc:824] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:14.710109 1381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c: No bootstrap required, opened a new log
Jun 20 21:50:14 dist-test-slave-kk5c krb5kdc[1300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:14.714438 1381 raft_consensus.cc:348] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f322f137f4a44cccaed3c52684a64d0c" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38729 } }
I20260620 21:50:14.714741 1381 raft_consensus.cc:374] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:14.714814 1381 raft_consensus.cc:729] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f322f137f4a44cccaed3c52684a64d0c, State: Initialized, Role: FOLLOWER
I20260620 21:50:14.715410 1381 consensus_queue.cc:260] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f322f137f4a44cccaed3c52684a64d0c" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38729 } }
I20260620 21:50:14.715583 1381 raft_consensus.cc:388] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:14.715677 1381 raft_consensus.cc:482] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:14.715785 1381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:14.716856 1381 raft_consensus.cc:504] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f322f137f4a44cccaed3c52684a64d0c" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38729 } }
I20260620 21:50:14.717288 1381 leader_election.cc:302] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f322f137f4a44cccaed3c52684a64d0c; no voters:
I20260620 21:50:14.717677 1381 leader_election.cc:288] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:14.717774 1386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:14.717988 1386 raft_consensus.cc:686] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [term 1 LEADER]: Becoming Leader. State: Replica: f322f137f4a44cccaed3c52684a64d0c, State: Running, Role: LEADER
I20260620 21:50:14.718255 1384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:14.702936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.0.193.254:38729)
0620 21:50:14.703463 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:14.703479 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:14.703516 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:50:14.704208 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:14.704227 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:14.704581 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:14.704983 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:14.705800 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:14.706827 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:14.707813 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:14.708503 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:14.708685 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:14.711356 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:14.711361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:14.711376 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:14.711406 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:14.713589 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:14.714467 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:14.714473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:14.714479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:14.714622 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:14.715149 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:14.715153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:14.715155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:14.715926 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:14.716172 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:14.716601 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 21:50:14.716939 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":174,"threads_started":1}
I20260620 21:50:14.718369 1386 consensus_queue.cc:237] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f322f137f4a44cccaed3c52684a64d0c" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38729 } }
I20260620 21:50:14.719089 1381 sys_catalog.cc:564] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:14.719719 1386 sys_catalog.cc:455] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f322f137f4a44cccaed3c52684a64d0c. Latest consensus state: current_term: 1 leader_uuid: "f322f137f4a44cccaed3c52684a64d0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f322f137f4a44cccaed3c52684a64d0c" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38729 } } }
I20260620 21:50:14.719937 1386 sys_catalog.cc:458] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:14.720218 1387 sys_catalog.cc:455] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f322f137f4a44cccaed3c52684a64d0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f322f137f4a44cccaed3c52684a64d0c" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38729 } } }
I20260620 21:50:14.720340 1387 sys_catalog.cc:458] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:14.721706 1394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:14.725473 1394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:14.731835 1394 catalog_manager.cc:1347] Generated new cluster ID: 45fc338ca633436788e3bd7286153485
I20260620 21:50:14.731928 1394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:14.748072 1394 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:14.749432 1394 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:14.756529 1394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f322f137f4a44cccaed3c52684a64d0c: Generated new TSK 0
I20260620 21:50:14.757524 1394 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:50:14.795984 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38729
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:14.927380 1407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:14.927706 1407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:14.927760 1407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:14.932648 1407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:14.932798 1407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:14.932940 1407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:14.939239 1407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:14.940487 1407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:14.941691 1407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:14.952239 1412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:14.952248 1413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:14.952296 1407 server_base.cc:1034] running on GCE node
W20260620 21:50:14.952248 1415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:14.953266 1407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:14.954025 1407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:14.955284 1407 hybrid_clock.cc:648] HybridClock initialized: now 1781992214955266 us; error 36 us; skew 500 ppm
Jun 20 21:50:14 dist-test-slave-kk5c krb5kdc[1300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:14.958428 1407 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:50:14.959503 1407 webserver.cc:460] Webserver started at http://127.0.193.193:34013/ using document root <none> and password file <none>
I20260620 21:50:14.960110 1407 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:14.960204 1407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:14.960446 1407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:14.962584 1407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "d84825efb167441a8dc8754da29c286a"
format_stamp: "Formatted at 2026-06-20 21:50:14 on dist-test-slave-kk5c"
server_key: "5f229309d2941644b591a18b58fa75b3"
server_key_iv: "80fd23edacaed61c3cc393b5427d7723"
server_key_version: "encryptionkey@0"
I20260620 21:50:14.963251 1407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "d84825efb167441a8dc8754da29c286a"
format_stamp: "Formatted at 2026-06-20 21:50:14 on dist-test-slave-kk5c"
server_key: "5f229309d2941644b591a18b58fa75b3"
server_key_iv: "80fd23edacaed61c3cc393b5427d7723"
server_key_version: "encryptionkey@0"
I20260620 21:50:14.967362 1407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:50:14.970034 1422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:14.971590 1407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:14.971762 1407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "d84825efb167441a8dc8754da29c286a"
format_stamp: "Formatted at 2026-06-20 21:50:14 on dist-test-slave-kk5c"
server_key: "5f229309d2941644b591a18b58fa75b3"
server_key_iv: "80fd23edacaed61c3cc393b5427d7723"
server_key_version: "encryptionkey@0"
I20260620 21:50:14.971900 1407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:14.994856 1407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:14.998461 1407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:14.998687 1407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:14.999387 1407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:15.000330 1407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:15.000404 1407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:15.000478 1407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:15.000530 1407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:15.011884 1407 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:36985
I20260620 21:50:15.012115 1535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:36985 every 8 connection(s)
I20260620 21:50:15.012939 1407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:15.014420 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 1407
I20260620 21:50:15.014529 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:50:15.014799 775 external_mini_cluster.cc:1428] Setting key 7508b923f8be3c6e9fbb8ba172d05f99
Jun 20 21:50:15 dist-test-slave-kk5c krb5kdc[1300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
I20260620 21:50:15.031666 1384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.015913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:49627 (local address 127.0.193.254:38729)
0620 21:50:15.016044 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:15.016055 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:15.017245 (+ 1190us) server_negotiation.cc:407] Connection header received
0620 21:50:15.018697 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:15.018702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:15.018776 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:15.018886 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:15.020415 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.021311 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.022670 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.023151 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.023305 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.026782 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:15.026786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:15.026788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:15.026825 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:15.028732 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:15.029706 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:15.029709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:15.029711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:15.029796 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:15.030328 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:15.030331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:15.030332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:15.030548 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:15.030717 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:15.031345 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 21:50:15.031490 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:50:15.032631 1538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.016492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:38729 (local address 127.0.193.193:49627)
0620 21:50:15.017061 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:15.017118 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:15.018421 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:15.019056 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:15.019064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:15.019698 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:15.020241 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.020256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.021512 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.021520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.022515 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.022525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.023409 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.023412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.023592 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.024519 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:15.024538 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:15.026549 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:15.028940 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:15.028987 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:15.029012 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:15.029565 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:15.029903 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:15.029910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:15.029915 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:15.030157 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:15.030717 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:15.030726 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:15.031030 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 21:50:15.031303 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 21:50:15.031690 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":174,"threads_started":1}
I20260620 21:50:15.034024 1536 heartbeater.cc:347] Connected to a master server at 127.0.193.254:38729
I20260620 21:50:15.034334 1536 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:15.034870 1536 heartbeater.cc:511] Master 127.0.193.254:38729 requested a full tablet report, sending...
I20260620 21:50:15.036623 1344 ts_manager.cc:194] Registered new tserver with Master: d84825efb167441a8dc8754da29c286a (127.0.193.193:36985)
I20260620 21:50:15.037792 1344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:49627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:50:15.047042 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38729
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:15.163488 1541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:15.163764 1541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:15.163811 1541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:15.168426 1541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:15.168511 1541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:15.168645 1541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:15.174454 1541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:15.175717 1541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:15.176916 1541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:15.187887 1546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:15.188097 1541 server_base.cc:1034] running on GCE node
W20260620 21:50:15.187899 1549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:15.187887 1547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:15.189111 1541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:15.189957 1541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:15.191268 1541 hybrid_clock.cc:648] HybridClock initialized: now 1781992215191211 us; error 65 us; skew 500 ppm
Jun 20 21:50:15 dist-test-slave-kk5c krb5kdc[1300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:15.194622 1541 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:50:15.195698 1541 webserver.cc:460] Webserver started at http://127.0.193.194:33379/ using document root <none> and password file <none>
I20260620 21:50:15.196310 1541 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:15.196379 1541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:15.196619 1541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:15.198873 1541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "b229108c88534f909717ca3af08b32ed"
format_stamp: "Formatted at 2026-06-20 21:50:15 on dist-test-slave-kk5c"
server_key: "a5ad81ac22494059fbd09d105a25d5c1"
server_key_iv: "327f5ab800327827f9aa0140a7a4d9df"
server_key_version: "encryptionkey@0"
I20260620 21:50:15.199568 1541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "b229108c88534f909717ca3af08b32ed"
format_stamp: "Formatted at 2026-06-20 21:50:15 on dist-test-slave-kk5c"
server_key: "a5ad81ac22494059fbd09d105a25d5c1"
server_key_iv: "327f5ab800327827f9aa0140a7a4d9df"
server_key_version: "encryptionkey@0"
I20260620 21:50:15.203881 1541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:50:15.206483 1556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:15.207926 1541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:15.208072 1541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "b229108c88534f909717ca3af08b32ed"
format_stamp: "Formatted at 2026-06-20 21:50:15 on dist-test-slave-kk5c"
server_key: "a5ad81ac22494059fbd09d105a25d5c1"
server_key_iv: "327f5ab800327827f9aa0140a7a4d9df"
server_key_version: "encryptionkey@0"
I20260620 21:50:15.208197 1541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:15.231027 1541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:15.234812 1541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:15.235060 1541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:15.235780 1541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:15.236691 1541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:15.236769 1541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:15.236840 1541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:15.236893 1541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:15.246562 1541 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:42805
I20260620 21:50:15.246624 1669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:42805 every 8 connection(s)
I20260620 21:50:15.247601 1541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:15.254431 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 1541
I20260620 21:50:15.254565 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:50:15.254841 775 external_mini_cluster.cc:1428] Setting key 8f87ab8608636a73d1fab73a700fffeb
Jun 20 21:50:15 dist-test-slave-kk5c krb5kdc[1300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:15.263407 1384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.249547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:41995 (local address 127.0.193.254:38729)
0620 21:50:15.249777 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:15.249782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:15.250648 (+ 866us) server_negotiation.cc:407] Connection header received
0620 21:50:15.252141 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:15.252143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:15.252193 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:15.252271 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:15.253512 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.254076 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.255495 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.255799 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.255923 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.258977 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:15.258981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:15.258983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:15.259009 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:15.260684 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:15.261250 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:15.261254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:15.261255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:15.261558 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:15.262028 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:15.262031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:15.262032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:15.262252 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:15.262370 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:15.262952 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 21:50:15.263095 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
I20260620 21:50:15.264525 1672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.249990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:38729 (local address 127.0.193.194:41995)
0620 21:50:15.250478 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:15.250524 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:15.251886 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:15.252439 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:15.252446 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:15.252826 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:15.253371 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.253384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.254227 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.254230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.255375 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.255388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.255976 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.255979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.256208 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.256983 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:15.257000 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:15.258779 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:15.260813 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:15.260820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:15.260839 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:15.261135 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:15.261692 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:15.261698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:15.261701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:15.261889 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:15.262370 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:15.262379 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:15.262691 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 21:50:15.262899 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 21:50:15.263422 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":131,"threads_started":1}
I20260620 21:50:15.265949 1670 heartbeater.cc:347] Connected to a master server at 127.0.193.254:38729
I20260620 21:50:15.266230 1670 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:15.266824 1670 heartbeater.cc:511] Master 127.0.193.254:38729 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
I20260620 21:50:15.268239 1344 ts_manager.cc:194] Registered new tserver with Master: b229108c88534f909717ca3af08b32ed (127.0.193.194:42805)
I20260620 21:50:15.268905 1344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:41995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:50:15.288771 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38729
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:41057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:15.423790 1675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:15.424073 1675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:15.424126 1675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:15.429314 1675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:15.429399 1675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:15.429463 1675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:15.435336 1675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:41057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:15.436653 1675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:15.437798 1675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:15.449863 1683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:15.449944 1675 server_base.cc:1034] running on GCE node
W20260620 21:50:15.449837 1680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:15.449872 1681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:15.450625 1675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:15.451368 1675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:15.452641 1675 hybrid_clock.cc:648] HybridClock initialized: now 1781992215452606 us; error 52 us; skew 500 ppm
Jun 20 21:50:15 dist-test-slave-kk5c krb5kdc[1300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:15.455814 1675 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:50:15.456831 1675 webserver.cc:460] Webserver started at http://127.0.193.195:37007/ using document root <none> and password file <none>
I20260620 21:50:15.457417 1675 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:15.457504 1675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:15.457724 1675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:15.459954 1675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "2980ef79aae2445fa95fc35607ba7a1c"
format_stamp: "Formatted at 2026-06-20 21:50:15 on dist-test-slave-kk5c"
server_key: "701d1daf6bd03360b9ff87f0cb0a27f1"
server_key_iv: "ade1547a7168a61a797e4edd43f62d73"
server_key_version: "encryptionkey@0"
I20260620 21:50:15.460546 1675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "2980ef79aae2445fa95fc35607ba7a1c"
format_stamp: "Formatted at 2026-06-20 21:50:15 on dist-test-slave-kk5c"
server_key: "701d1daf6bd03360b9ff87f0cb0a27f1"
server_key_iv: "ade1547a7168a61a797e4edd43f62d73"
server_key_version: "encryptionkey@0"
I20260620 21:50:15.464777 1675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:50:15.468473 1690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:15.469897 1675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:50:15.470063 1675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "2980ef79aae2445fa95fc35607ba7a1c"
format_stamp: "Formatted at 2026-06-20 21:50:15 on dist-test-slave-kk5c"
server_key: "701d1daf6bd03360b9ff87f0cb0a27f1"
server_key_iv: "ade1547a7168a61a797e4edd43f62d73"
server_key_version: "encryptionkey@0"
I20260620 21:50:15.470196 1675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:15.484283 1675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:15.487777 1675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:15.487996 1675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:15.488654 1675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:15.489531 1675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:15.489604 1675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:15.489681 1675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:15.489722 1675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:15.499610 1675 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:39547
I20260620 21:50:15.499652 1803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:39547 every 8 connection(s)
I20260620 21:50:15.500576 1675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:50:15.506557 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 1675
I20260620 21:50:15.506793 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:50:15.507115 775 external_mini_cluster.cc:1428] Setting key 5a37378541fa194a93d5addae1200ddb
Jun 20 21:50:15 dist-test-slave-kk5c krb5kdc[1300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:15.516917 1384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.502715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:39181 (local address 127.0.193.254:38729)
0620 21:50:15.502881 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:15.502885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:15.504249 (+ 1364us) server_negotiation.cc:407] Connection header received
0620 21:50:15.505193 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:15.505196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:15.505246 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:15.505327 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:15.506690 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.507574 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.509162 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.509425 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.509507 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.512373 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:15.512377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:15.512379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:15.512408 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:15.514384 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:15.515106 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:15.515110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:15.515112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:15.515282 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:15.515718 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:15.515726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:15.515728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:15.515968 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:15.516061 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:15.516657 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 21:50:15.516787 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 21:50:15.517880 1806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.503121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:38729 (local address 127.0.193.195:39181)
0620 21:50:15.504091 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:15.504137 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:15.504957 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:15.505519 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:15.505526 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:15.505988 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:15.506506 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.506521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.507730 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.507734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.509004 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.509016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.509561 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.509564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.509732 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.510321 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:15.510339 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:15.512112 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:15.514538 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:15.514545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:15.514560 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:15.514939 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:15.515408 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:15.515410 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:15.515412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:15.515599 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:15.516109 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:15.516115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:15.516406 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 21:50:15.516583 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 21:50:15.516895 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":550,"threads_started":1}
I20260620 21:50:15.519461 1804 heartbeater.cc:347] Connected to a master server at 127.0.193.254:38729
I20260620 21:50:15.519822 1804 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:15.520478 1804 heartbeater.cc:511] Master 127.0.193.254:38729 requested a full tablet report, sending...
I20260620 21:50:15.521751 1344 ts_manager.cc:194] Registered new tserver with Master: 2980ef79aae2445fa95fc35607ba7a1c (127.0.193.195:39547)
I20260620 21:50:15.522403 1344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:39181
I20260620 21:50:15.534132 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:50:15.546249 1384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.537996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51300 (local address 127.0.193.254:38729)
0620 21:50:15.538142 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:15.538149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:15.538352 (+ 203us) server_negotiation.cc:407] Connection header received
0620 21:50:15.538438 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:15.538441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:15.538490 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:15.538574 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:15.539323 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.539919 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.540748 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.540987 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.541122 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.542510 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:15.542513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:15.542515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:15.542544 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:15.544183 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:15.544631 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:15.544634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:15.544636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:15.544770 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:15.545058 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:15.545061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:15.545063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:15.545351 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:15.545484 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:15.545856 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 21:50:15.546006 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:50:15.553418 1344 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:50:15.557039 1344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:15.572871 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.566203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.0.193.195:39547)
0620 21:50:15.566535 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:15.566550 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:15.566650 (+ 100us) server_negotiation.cc:407] Connection header received
0620 21:50:15.566956 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:15.566963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:15.567261 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:15.567432 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:15.567985 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.569005 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.570842 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.571813 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.571901 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.572058 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:15.572579 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 21:50:15.572686 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":109,"threads_started":1}
I20260620 21:50:15.572882 1815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.566005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:39547 (local address 127.0.0.1:56198)
0620 21:50:15.566482 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:15.566542 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:15.566729 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:15.567656 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:15.567663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:15.567689 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:15.567840 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.567851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.569147 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.569153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.570687 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.570698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.572172 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.572176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.572310 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.572355 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 21:50:15.572529 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 21:50:15.572692 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":3,"thread_start_us":152,"threads_started":1}
I20260620 21:50:15.573419 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.566931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:42805 (local address 127.0.0.1:52760)
0620 21:50:15.567833 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:15.567850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:15.567949 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:15.568489 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:15.568493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:15.568513 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:15.568618 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.568625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.569845 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.569849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.571154 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.571164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.572331 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.572335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.572434 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.572461 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:50:15.572548 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 21:50:15.573280 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":646,"thread_start_us":104,"threads_started":1}
I20260620 21:50:15.572953 1818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.567103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.0.193.194:42805)
0620 21:50:15.567668 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:15.567685 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:15.568077 (+ 392us) server_negotiation.cc:407] Connection header received
0620 21:50:15.568169 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:15.568174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:15.568338 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:15.568470 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:15.568736 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.569684 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.571321 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.572201 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.572311 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.572455 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:15.572672 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 21:50:15.572782 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":202,"threads_started":1}
I20260620 21:50:15.574911 1817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.567330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:36985 (local address 127.0.0.1:40312)
0620 21:50:15.569280 (+ 1950us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:15.569295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:15.569390 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:15.569990 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:15.569992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:15.570008 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:15.570180 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.570188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.571587 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.571594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.572548 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.572557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.574458 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.574463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.574564 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.574589 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:50:15.574699 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 21:50:15.574776 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1879,"mutex_wait_us":61,"thread_start_us":85,"threads_started":1}
I20260620 21:50:15.575344 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.567468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40312 (local address 127.0.193.193:36985)
0620 21:50:15.567775 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:15.567783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:15.569406 (+ 1623us) server_negotiation.cc:407] Connection header received
0620 21:50:15.569494 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:15.569499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:15.569703 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:15.569853 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:15.570328 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.571474 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.572699 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.574027 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.574245 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.574395 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:15.574750 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 21:50:15.574899 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
I20260620 21:50:15.576504 1604 tablet_service.cc:1467] Processing CreateTablet for tablet 3504e56b46cd41d088365c85381e4eea (DEFAULT_TABLE table=test-table [id=3ec43cb69d4648bd8b51d0a860fe26db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:15.576503 1738 tablet_service.cc:1467] Processing CreateTablet for tablet 3504e56b46cd41d088365c85381e4eea (DEFAULT_TABLE table=test-table [id=3ec43cb69d4648bd8b51d0a860fe26db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:15.577543 1604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3504e56b46cd41d088365c85381e4eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:15.577543 1738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3504e56b46cd41d088365c85381e4eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:15.582535 1470 tablet_service.cc:1467] Processing CreateTablet for tablet 3504e56b46cd41d088365c85381e4eea (DEFAULT_TABLE table=test-table [id=3ec43cb69d4648bd8b51d0a860fe26db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:15.583923 1470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3504e56b46cd41d088365c85381e4eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:15.584856 1821 tablet_bootstrap.cc:492] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c: Bootstrap starting.
I20260620 21:50:15.585736 1822 tablet_bootstrap.cc:492] T 3504e56b46cd41d088365c85381e4eea P b229108c88534f909717ca3af08b32ed: Bootstrap starting.
I20260620 21:50:15.587642 1821 tablet_bootstrap.cc:654] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:15.588495 1821 log.cc:824] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:15.588801 1822 tablet_bootstrap.cc:654] T 3504e56b46cd41d088365c85381e4eea P b229108c88534f909717ca3af08b32ed: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:15.589747 1822 log.cc:824] T 3504e56b46cd41d088365c85381e4eea P b229108c88534f909717ca3af08b32ed: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:15.591109 1821 tablet_bootstrap.cc:492] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c: No bootstrap required, opened a new log
I20260620 21:50:15.591363 1821 ts_tablet_manager.cc:1397] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260620 21:50:15.592568 1822 tablet_bootstrap.cc:492] T 3504e56b46cd41d088365c85381e4eea P b229108c88534f909717ca3af08b32ed: No bootstrap required, opened a new log
I20260620 21:50:15.592784 1822 ts_tablet_manager.cc:1397] T 3504e56b46cd41d088365c85381e4eea P b229108c88534f909717ca3af08b32ed: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 21:50:15.593331 1825 tablet_bootstrap.cc:492] T 3504e56b46cd41d088365c85381e4eea P d84825efb167441a8dc8754da29c286a: Bootstrap starting.
I20260620 21:50:15.595431 1821 raft_consensus.cc:348] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2980ef79aae2445fa95fc35607ba7a1c" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39547 } } peers { permanent_uuid: "b229108c88534f909717ca3af08b32ed" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 42805 } } peers { permanent_uuid: "d84825efb167441a8dc8754da29c286a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 36985 } }
I20260620 21:50:15.595763 1821 raft_consensus.cc:374] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:15.595824 1821 raft_consensus.cc:729] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2980ef79aae2445fa95fc35607ba7a1c, State: Initialized, Role: FOLLOWER
I20260620 21:50:15.596310 1821 consensus_queue.cc:260] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2980ef79aae2445fa95fc35607ba7a1c" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39547 } } peers { permanent_uuid: "b229108c88534f909717ca3af08b32ed" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 42805 } } peers { permanent_uuid: "d84825efb167441a8dc8754da29c286a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 36985 } }
I20260620 21:50:15.596308 1822 raft_consensus.cc:348] T 3504e56b46cd41d088365c85381e4eea P b229108c88534f909717ca3af08b32ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2980ef79aae2445fa95fc35607ba7a1c" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39547 } } peers { permanent_uuid: "b229108c88534f909717ca3af08b32ed" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 42805 } } peers { permanent_uuid: "d84825efb167441a8dc8754da29c286a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 36985 } }
I20260620 21:50:15.596601 1822 raft_consensus.cc:374] T 3504e56b46cd41d088365c85381e4eea P b229108c88534f909717ca3af08b32ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:15.596684 1822 raft_consensus.cc:729] T 3504e56b46cd41d088365c85381e4eea P b229108c88534f909717ca3af08b32ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b229108c88534f909717ca3af08b32ed, State: Initialized, Role: FOLLOWER
I20260620 21:50:15.597208 1825 tablet_bootstrap.cc:654] T 3504e56b46cd41d088365c85381e4eea P d84825efb167441a8dc8754da29c286a: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:15.597229 1804 heartbeater.cc:503] Master 127.0.193.254:38729 was elected leader, sending a full tablet report...
I20260620 21:50:15.597155 1822 consensus_queue.cc:260] T 3504e56b46cd41d088365c85381e4eea P b229108c88534f909717ca3af08b32ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2980ef79aae2445fa95fc35607ba7a1c" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39547 } } peers { permanent_uuid: "b229108c88534f909717ca3af08b32ed" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 42805 } } peers { permanent_uuid: "d84825efb167441a8dc8754da29c286a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 36985 } }
I20260620 21:50:15.597474 1821 ts_tablet_manager.cc:1428] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260620 21:50:15.598156 1822 ts_tablet_manager.cc:1428] T 3504e56b46cd41d088365c85381e4eea P b229108c88534f909717ca3af08b32ed: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 21:50:15.598196 1825 log.cc:824] T 3504e56b46cd41d088365c85381e4eea P d84825efb167441a8dc8754da29c286a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:15.598520 1670 heartbeater.cc:503] Master 127.0.193.254:38729 was elected leader, sending a full tablet report...
I20260620 21:50:15.601222 1825 tablet_bootstrap.cc:492] T 3504e56b46cd41d088365c85381e4eea P d84825efb167441a8dc8754da29c286a: No bootstrap required, opened a new log
I20260620 21:50:15.601471 1825 ts_tablet_manager.cc:1397] T 3504e56b46cd41d088365c85381e4eea P d84825efb167441a8dc8754da29c286a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 21:50:15.605564 1825 raft_consensus.cc:348] T 3504e56b46cd41d088365c85381e4eea P d84825efb167441a8dc8754da29c286a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2980ef79aae2445fa95fc35607ba7a1c" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39547 } } peers { permanent_uuid: "b229108c88534f909717ca3af08b32ed" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 42805 } } peers { permanent_uuid: "d84825efb167441a8dc8754da29c286a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 36985 } }
I20260620 21:50:15.605852 1825 raft_consensus.cc:374] T 3504e56b46cd41d088365c85381e4eea P d84825efb167441a8dc8754da29c286a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:15.605898 1825 raft_consensus.cc:729] T 3504e56b46cd41d088365c85381e4eea P d84825efb167441a8dc8754da29c286a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84825efb167441a8dc8754da29c286a, State: Initialized, Role: FOLLOWER
I20260620 21:50:15.606338 1825 consensus_queue.cc:260] T 3504e56b46cd41d088365c85381e4eea P d84825efb167441a8dc8754da29c286a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2980ef79aae2445fa95fc35607ba7a1c" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39547 } } peers { permanent_uuid: "b229108c88534f909717ca3af08b32ed" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 42805 } } peers { permanent_uuid: "d84825efb167441a8dc8754da29c286a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 36985 } }
I20260620 21:50:15.607163 1536 heartbeater.cc:503] Master 127.0.193.254:38729 was elected leader, sending a full tablet report...
I20260620 21:50:15.607568 1825 ts_tablet_manager.cc:1428] T 3504e56b46cd41d088365c85381e4eea P d84825efb167441a8dc8754da29c286a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:50:15.700783 1826 raft_consensus.cc:482] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:15.701035 1826 raft_consensus.cc:504] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2980ef79aae2445fa95fc35607ba7a1c" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39547 } } peers { permanent_uuid: "b229108c88534f909717ca3af08b32ed" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 42805 } } peers { permanent_uuid: "d84825efb167441a8dc8754da29c286a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 36985 } }
I20260620 21:50:15.702494 1826 leader_election.cc:288] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b229108c88534f909717ca3af08b32ed (127.0.193.194:42805), d84825efb167441a8dc8754da29c286a (127.0.193.193:36985)
I20260620 21:50:15.708487 1818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.702923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:36467 (local address 127.0.193.194:42805)
0620 21:50:15.703074 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:15.703080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:15.703097 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:50:15.703324 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:15.703331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:15.703394 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:15.703461 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:15.703887 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.704665 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.707424 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.707829 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.707972 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.708032 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:15.708256 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 21:50:15.708355 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:50:15.708515 1806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.702833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:42805 (local address 127.0.193.195:36467)
0620 21:50:15.703001 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:15.703020 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:15.703135 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:15.703597 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:15.703603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:15.703625 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:15.703768 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.703777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.705346 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.705351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.707241 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.707262 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.707962 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.707966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.708072 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.708128 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 21:50:15.708205 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 21:50:15.708307 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":2}
I20260620 21:50:15.708534 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.703368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:34551 (local address 127.0.193.193:36985)
0620 21:50:15.703504 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:15.703510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:15.704331 (+ 821us) server_negotiation.cc:407] Connection header received
0620 21:50:15.704592 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:15.704597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:15.704662 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:15.704740 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:15.705115 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.705700 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.707549 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.707980 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.708128 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.708181 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:15.708360 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 21:50:15.708432 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:50:15.708905 1830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.703119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:36985 (local address 127.0.193.195:34551)
0620 21:50:15.704234 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:15.704251 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:15.704401 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:15.704773 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:15.704779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:15.704806 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:15.704977 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.704986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.705813 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.705819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.707412 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:15.707423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.708109 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.708112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:15.708224 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.708248 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:50:15.708665 (+ 417us) client_negotiation.cc:240] Negotiation successful
0620 21:50:15.708752 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"mutex_wait_us":192,"thread_start_us":898,"threads_started":1}
I20260620 21:50:15.709372 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3504e56b46cd41d088365c85381e4eea" candidate_uuid: "2980ef79aae2445fa95fc35607ba7a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b229108c88534f909717ca3af08b32ed" is_pre_election: true
I20260620 21:50:15.709558 1490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3504e56b46cd41d088365c85381e4eea" candidate_uuid: "2980ef79aae2445fa95fc35607ba7a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84825efb167441a8dc8754da29c286a" is_pre_election: true
I20260620 21:50:15.709769 1624 raft_consensus.cc:2454] T 3504e56b46cd41d088365c85381e4eea P b229108c88534f909717ca3af08b32ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2980ef79aae2445fa95fc35607ba7a1c in term 0.
I20260620 21:50:15.709862 1490 raft_consensus.cc:2454] T 3504e56b46cd41d088365c85381e4eea P d84825efb167441a8dc8754da29c286a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2980ef79aae2445fa95fc35607ba7a1c in term 0.
I20260620 21:50:15.710296 1693 leader_election.cc:302] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2980ef79aae2445fa95fc35607ba7a1c, b229108c88534f909717ca3af08b32ed; no voters:
I20260620 21:50:15.710649 1826 raft_consensus.cc:2781] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:15.710760 1826 raft_consensus.cc:482] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:15.710817 1826 raft_consensus.cc:3037] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:15.712512 1826 raft_consensus.cc:504] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2980ef79aae2445fa95fc35607ba7a1c" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39547 } } peers { permanent_uuid: "b229108c88534f909717ca3af08b32ed" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 42805 } } peers { permanent_uuid: "d84825efb167441a8dc8754da29c286a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 36985 } }
I20260620 21:50:15.712839 1826 leader_election.cc:288] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [CANDIDATE]: Term 1 election: Requested vote from peers b229108c88534f909717ca3af08b32ed (127.0.193.194:42805), d84825efb167441a8dc8754da29c286a (127.0.193.193:36985)
I20260620 21:50:15.713316 1624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3504e56b46cd41d088365c85381e4eea" candidate_uuid: "2980ef79aae2445fa95fc35607ba7a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b229108c88534f909717ca3af08b32ed"
I20260620 21:50:15.713316 1490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3504e56b46cd41d088365c85381e4eea" candidate_uuid: "2980ef79aae2445fa95fc35607ba7a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84825efb167441a8dc8754da29c286a"
I20260620 21:50:15.713455 1624 raft_consensus.cc:3037] T 3504e56b46cd41d088365c85381e4eea P b229108c88534f909717ca3af08b32ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:15.713464 1490 raft_consensus.cc:3037] T 3504e56b46cd41d088365c85381e4eea P d84825efb167441a8dc8754da29c286a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:15.714769 1490 raft_consensus.cc:2454] T 3504e56b46cd41d088365c85381e4eea P d84825efb167441a8dc8754da29c286a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2980ef79aae2445fa95fc35607ba7a1c in term 1.
I20260620 21:50:15.714915 1624 raft_consensus.cc:2454] T 3504e56b46cd41d088365c85381e4eea P b229108c88534f909717ca3af08b32ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2980ef79aae2445fa95fc35607ba7a1c in term 1.
I20260620 21:50:15.715252 1693 leader_election.cc:302] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2980ef79aae2445fa95fc35607ba7a1c, b229108c88534f909717ca3af08b32ed; no voters:
I20260620 21:50:15.715541 1826 raft_consensus.cc:2781] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:15.715746 1826 raft_consensus.cc:686] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [term 1 LEADER]: Becoming Leader. State: Replica: 2980ef79aae2445fa95fc35607ba7a1c, State: Running, Role: LEADER
I20260620 21:50:15.716118 1826 consensus_queue.cc:237] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2980ef79aae2445fa95fc35607ba7a1c" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39547 } } peers { permanent_uuid: "b229108c88534f909717ca3af08b32ed" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 42805 } } peers { permanent_uuid: "d84825efb167441a8dc8754da29c286a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 36985 } }
I20260620 21:50:15.719839 1344 catalog_manager.cc:5495] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2980ef79aae2445fa95fc35607ba7a1c (127.0.193.195). New cstate: current_term: 1 leader_uuid: "2980ef79aae2445fa95fc35607ba7a1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2980ef79aae2445fa95fc35607ba7a1c" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b229108c88534f909717ca3af08b32ed" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d84825efb167441a8dc8754da29c286a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 36985 } health_report { overall_health: UNKNOWN } } }
W20260620 21:50:15.748948 1671 tablet.cc:2369] T 3504e56b46cd41d088365c85381e4eea P b229108c88534f909717ca3af08b32ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:50:15.751572 1805 tablet.cc:2369] T 3504e56b46cd41d088365c85381e4eea P 2980ef79aae2445fa95fc35607ba7a1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 21:50:15 dist-test-slave-kk5c krb5kdc[1300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.193@KRBTEST.COM
W20260620 21:50:15.764581 1537 tablet.cc:2369] T 3504e56b46cd41d088365c85381e4eea P d84825efb167441a8dc8754da29c286a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:50:15.768151 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.757630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.0.193.193:36985)
0620 21:50:15.757802 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:15.757808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:15.757887 (+ 79us) server_negotiation.cc:407] Connection header received
0620 21:50:15.758049 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:15.758052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:15.758101 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:15.758200 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:15.758973 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.759560 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.760412 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.760628 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.760737 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.763729 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:15.763742 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:15.763757 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:15.763790 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:15.765928 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:15.766489 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:15.766496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:15.766497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:15.766641 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:15.766946 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:15.766948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:15.766950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:15.767326 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:15.767515 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:15.767871 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 21:50:15.767993 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 21:50:15 dist-test-slave-kk5c krb5kdc[1300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:50:15 dist-test-slave-kk5c krb5kdc[1300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.193.193@KRBTEST.COM
I20260620 21:50:15.800740 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.789863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40318 (local address 127.0.193.193:36985)
0620 21:50:15.790018 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:15.790023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:15.790294 (+ 271us) server_negotiation.cc:407] Connection header received
0620 21:50:15.790383 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:15.790385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:15.790434 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:15.790522 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:15.792043 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.793037 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.794290 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.794560 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.794663 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.797422 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:15.797426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:15.797428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:15.797460 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:15.798964 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:15.799492 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:15.799495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:15.799497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:15.799592 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:15.799821 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:15.799823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:15.799825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:15.800005 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:15.800144 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:15.800373 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 21:50:15.800529 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 21:50:15 dist-test-slave-kk5c krb5kdc[1300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:50:15 dist-test-slave-kk5c krb5kdc[1300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.193@KRBTEST.COM
I20260620 21:50:15.832193 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:15.821393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.0.193.193:36985)
0620 21:50:15.821589 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:15.821594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:15.821806 (+ 212us) server_negotiation.cc:407] Connection header received
0620 21:50:15.821987 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:15.821990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:15.822074 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:15.822168 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:15.822900 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.823488 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.824411 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:15.824638 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:15.824748 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:15.827764 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:15.827768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:15.827770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:15.827800 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:15.829942 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:15.830482 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:15.830485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:15.830487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:15.830583 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:15.830981 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:15.830985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:15.830986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:15.831265 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:15.831392 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:15.831852 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 21:50:15.832034 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 21:50:15.833523 1450 tablet_service.cc:2908] Rejecting scan request for tablet 3504e56b46cd41d088365c85381e4eea: Uninitialized: clean time has not yet been initialized
I20260620 21:50:15.836091 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 1407
I20260620 21:50:15.843119 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 1541
I20260620 21:50:15.849611 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 1675
I20260620 21:50:15.855664 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 1313
2026-06-20T21:50:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2624 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:50:15 dist-test-slave-kk5c krb5kdc[1845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:50:15 dist-test-slave-kk5c krb5kdc[1845](info): set up 2 sockets
Jun 20 21:50:15 dist-test-slave-kk5c krb5kdc[1845](info): commencing operation
krb5kdc: starting...
W20260620 21:50:16.999527 775 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:50:17 dist-test-slave-kk5c krb5kdc[1845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:50:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:50:17.134532 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:34783
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:34783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:17.248425 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:17.248761 1858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:17.248843 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:17.253193 1858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:17.253298 1858 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 21:50:17.253329 1858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:17.253376 1858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:17.253422 1858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:17.253458 1858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:17.259081 1858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:34783
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:34783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:17.260331 1858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:17.261423 1858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:17.270460 1864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:17.270532 1863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:17.270668 1866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:17.270902 1858 server_base.cc:1034] running on GCE node
I20260620 21:50:17.271517 1858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:17.272665 1858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:17.273823 1858 hybrid_clock.cc:648] HybridClock initialized: now 1781992217273810 us; error 28 us; skew 500 ppm
Jun 20 21:50:17 dist-test-slave-kk5c krb5kdc[1845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:17.276742 1858 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:50:17.277686 1858 webserver.cc:460] Webserver started at http://127.0.193.254:36205/ using document root <none> and password file <none>
I20260620 21:50:17.278257 1858 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:17.278347 1858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:17.278573 1858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:17.280711 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "cd9dba5515ba4088a6075d6b0ed1fcc0"
format_stamp: "Formatted at 2026-06-20 21:50:17 on dist-test-slave-kk5c"
server_key: "ce989139c89950102904177c3ffdbd49"
server_key_iv: "ae2965b92ad2ab37e0242cf1ec501650"
server_key_version: "encryptionkey@0"
I20260620 21:50:17.281286 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "cd9dba5515ba4088a6075d6b0ed1fcc0"
format_stamp: "Formatted at 2026-06-20 21:50:17 on dist-test-slave-kk5c"
server_key: "ce989139c89950102904177c3ffdbd49"
server_key_iv: "ae2965b92ad2ab37e0242cf1ec501650"
server_key_version: "encryptionkey@0"
I20260620 21:50:17.285179 1858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:17.287808 1873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:17.289206 1858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:17.289361 1858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "cd9dba5515ba4088a6075d6b0ed1fcc0"
format_stamp: "Formatted at 2026-06-20 21:50:17 on dist-test-slave-kk5c"
server_key: "ce989139c89950102904177c3ffdbd49"
server_key_iv: "ae2965b92ad2ab37e0242cf1ec501650"
server_key_version: "encryptionkey@0"
I20260620 21:50:17.289481 1858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:17.312970 1858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:17.316468 1858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:17.316717 1858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:17.325021 1858 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:34783
I20260620 21:50:17.325050 1935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:34783 every 8 connection(s)
I20260620 21:50:17.325997 1858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:17.329452 1936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:17.331769 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 1858
I20260620 21:50:17.331902 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:17.332188 775 external_mini_cluster.cc:1428] Setting key e4b2bb13e2b37a3a032e3d5615d79763
I20260620 21:50:17.336566 1936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0: Bootstrap starting.
I20260620 21:50:17.339576 1936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:17.340559 1936 log.cc:824] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:50:17 dist-test-slave-kk5c krb5kdc[1845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:17.343369 1936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0: No bootstrap required, opened a new log
I20260620 21:50:17.348321 1936 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9dba5515ba4088a6075d6b0ed1fcc0" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 34783 } }
I20260620 21:50:17.348646 1936 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:17.348709 1936 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9dba5515ba4088a6075d6b0ed1fcc0, State: Initialized, Role: FOLLOWER
I20260620 21:50:17.349208 1936 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9dba5515ba4088a6075d6b0ed1fcc0" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 34783 } }
I20260620 21:50:17.349364 1936 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:17.349429 1936 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:17.349467 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:17.333723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42972 (local address 127.0.193.254:34783)
0620 21:50:17.334207 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:17.334232 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:17.334291 (+ 59us) server_negotiation.cc:407] Connection header received
0620 21:50:17.335278 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:17.335305 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:17.335800 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:17.336204 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:17.337136 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.338859 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.339941 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.340732 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.340916 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:17.343832 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:17.343841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:17.343867 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:17.343911 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:17.346042 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:17.346609 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:17.346615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:17.346622 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:17.346775 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:17.347058 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:17.347061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:17.347063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:17.347664 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:17.347860 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:17.348136 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 21:50:17.348450 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":123,"threads_started":1}
I20260620 21:50:17.349530 1936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:17.350513 1936 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9dba5515ba4088a6075d6b0ed1fcc0" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 34783 } }
I20260620 21:50:17.350885 1936 leader_election.cc:302] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd9dba5515ba4088a6075d6b0ed1fcc0; no voters:
I20260620 21:50:17.351259 1936 leader_election.cc:288] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:17.351478 1941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:17.351706 1941 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [term 1 LEADER]: Becoming Leader. State: Replica: cd9dba5515ba4088a6075d6b0ed1fcc0, State: Running, Role: LEADER
I20260620 21:50:17.352022 1941 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9dba5515ba4088a6075d6b0ed1fcc0" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 34783 } }
I20260620 21:50:17.352526 1936 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:17.353653 1943 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd9dba5515ba4088a6075d6b0ed1fcc0. Latest consensus state: current_term: 1 leader_uuid: "cd9dba5515ba4088a6075d6b0ed1fcc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9dba5515ba4088a6075d6b0ed1fcc0" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 34783 } } }
I20260620 21:50:17.353792 1943 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:17.354082 1942 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd9dba5515ba4088a6075d6b0ed1fcc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9dba5515ba4088a6075d6b0ed1fcc0" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 34783 } } }
I20260620 21:50:17.354193 1942 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:17.354142 1950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:17.357553 1950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:17.362931 1950 catalog_manager.cc:1347] Generated new cluster ID: 6b83039372394cf4be2cefdd211febcc
I20260620 21:50:17.363011 1950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:17.369823 1950 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:17.370808 1950 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:17.377084 1950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0: Generated new TSK 0
I20260620 21:50:17.377799 1950 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:50:17.410962 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:34783
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:17.535293 1962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:17.535601 1962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:17.535688 1962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:17.540203 1962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:17.540282 1962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:17.540323 1962 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:50:17.540393 1962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:17.546000 1962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.193.254:34783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:17.547307 1962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:17.548401 1962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:17.558743 1968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:17.558750 1967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:17.558740 1970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:17.559018 1962 server_base.cc:1034] running on GCE node
I20260620 21:50:17.559697 1962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:17.560387 1962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:17.561565 1962 hybrid_clock.cc:648] HybridClock initialized: now 1781992217561545 us; error 34 us; skew 500 ppm
Jun 20 21:50:17 dist-test-slave-kk5c krb5kdc[1845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:17.564613 1962 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:50:17.565609 1962 webserver.cc:460] Webserver started at http://127.0.193.193:38747/ using document root <none> and password file <none>
I20260620 21:50:17.566210 1962 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:17.566274 1962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:17.566547 1962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:17.569211 1962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "cb15b8c9da5a41198e80a18045f24345"
format_stamp: "Formatted at 2026-06-20 21:50:17 on dist-test-slave-kk5c"
server_key: "89d650bc02f24bda82fe3b816ebfccc1"
server_key_iv: "009fec6d6eb226f5deb6f36ac9e4540c"
server_key_version: "encryptionkey@0"
I20260620 21:50:17.569859 1962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "cb15b8c9da5a41198e80a18045f24345"
format_stamp: "Formatted at 2026-06-20 21:50:17 on dist-test-slave-kk5c"
server_key: "89d650bc02f24bda82fe3b816ebfccc1"
server_key_iv: "009fec6d6eb226f5deb6f36ac9e4540c"
server_key_version: "encryptionkey@0"
I20260620 21:50:17.574579 1962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 21:50:17.577963 1977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:17.579478 1962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:17.579627 1962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "cb15b8c9da5a41198e80a18045f24345"
format_stamp: "Formatted at 2026-06-20 21:50:17 on dist-test-slave-kk5c"
server_key: "89d650bc02f24bda82fe3b816ebfccc1"
server_key_iv: "009fec6d6eb226f5deb6f36ac9e4540c"
server_key_version: "encryptionkey@0"
I20260620 21:50:17.579761 1962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:17.614097 1962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:17.617923 1962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:17.618150 1962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:17.619814 1962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:17.619886 1962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:17.619957 1962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:17.619995 1962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:50:17 dist-test-slave-kk5c krb5kdc[1845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:17.633344 1962 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:38369
I20260620 21:50:17.633426 2092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:38369 every 8 connection(s)
I20260620 21:50:17.634471 1962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:17.635954 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:17.621403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:34783 (local address 127.0.193.193:51071)
0620 21:50:17.621993 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:17.622037 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:17.623136 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:17.623752 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:17.623759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:17.624103 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:17.624702 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:17.624719 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.625524 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.625528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:17.626499 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:17.626509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.627029 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.627033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:17.627257 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:17.628122 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:17.628149 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:17.630480 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:17.632680 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:17.632685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:17.632699 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:17.633015 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:17.633436 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:17.633440 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:17.633442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:17.633619 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:17.634038 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:17.634044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:17.634366 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 21:50:17.634588 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 21:50:17.634960 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":213,"threads_started":1}
I20260620 21:50:17.636777 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:17.620854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:51071 (local address 127.0.193.254:34783)
0620 21:50:17.620982 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:17.620985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:17.622181 (+ 1196us) server_negotiation.cc:407] Connection header received
0620 21:50:17.623406 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:17.623410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:17.623470 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:17.623569 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:17.624850 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.625390 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.626624 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.626862 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.626974 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:17.630738 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:17.630741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:17.630743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:17.630771 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:17.632532 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:17.633206 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:17.633210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:17.633211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:17.633324 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:17.633722 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:17.633725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:17.633726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:17.633932 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:17.634020 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:17.636437 (+ 2417us) server_negotiation.cc:299] Negotiation successful
0620 21:50:17.636647 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:50:17.637907 2093 heartbeater.cc:347] Connected to a master server at 127.0.193.254:34783
I20260620 21:50:17.638062 2093 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:17.638787 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 1962
I20260620 21:50:17.638769 2093 heartbeater.cc:511] Master 127.0.193.254:34783 requested a full tablet report, sending...
I20260620 21:50:17.638916 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:50:17.639359 775 external_mini_cluster.cc:1428] Setting key a3fc7a9628d861f0a8d411ab4495e6eb
I20260620 21:50:17.640563 1889 ts_manager.cc:194] Registered new tserver with Master: cb15b8c9da5a41198e80a18045f24345 (127.0.193.193:38369)
I20260620 21:50:17.641923 1889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:51071
I20260620 21:50:17.648922 2099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:17.640473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:34783 (local address 127.0.193.193:41347)
0620 21:50:17.640825 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:17.640840 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:17.640950 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:17.641203 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:17.641206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:17.641450 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:17.641643 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:17.641654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.642587 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.642590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:17.643313 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:17.643322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.643952 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.643956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:17.644059 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:17.644735 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:17.644747 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:17.645146 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:17.647366 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:17.647369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:17.647371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:17.647585 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:17.647909 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:17.647911 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:17.647913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:17.647996 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:17.648444 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:17.648447 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:17.648537 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 21:50:17.648640 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 21:50:17.648762 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
I20260620 21:50:17.649055 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:17.640518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:41347 (local address 127.0.193.254:34783)
0620 21:50:17.640642 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:17.640647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:17.640912 (+ 265us) server_negotiation.cc:407] Connection header received
0620 21:50:17.641033 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:17.641035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:17.641079 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:17.641162 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:17.641776 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.642468 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.643429 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.643814 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.643932 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:17.645300 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:17.645307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:17.645313 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:17.645346 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:17.647246 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:17.647696 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:17.647702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:17.647707 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:17.647810 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:17.648086 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:17.648091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:17.648095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:17.648339 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:17.648450 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:17.648715 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 21:50:17.648857 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:50:17.671454 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:34783
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:17.792843 2102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:17.793174 2102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:17.793263 2102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:17.797811 2102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:17.797911 2102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:17.797987 2102 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:50:17.798070 2102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:17.803807 2102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.193.254:34783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:17.804957 2102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:17.805992 2102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:17.815689 2110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:17.815780 2107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:17.815757 2102 server_base.cc:1034] running on GCE node
W20260620 21:50:17.815689 2108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:17.816342 2102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:17.816988 2102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:17.818197 2102 hybrid_clock.cc:648] HybridClock initialized: now 1781992217818172 us; error 37 us; skew 500 ppm
Jun 20 21:50:17 dist-test-slave-kk5c krb5kdc[1845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:17.821141 2102 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:50:17.822127 2102 webserver.cc:460] Webserver started at http://127.0.193.194:43703/ using document root <none> and password file <none>
I20260620 21:50:17.822677 2102 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:17.822738 2102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:17.822911 2102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:17.825086 2102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27"
format_stamp: "Formatted at 2026-06-20 21:50:17 on dist-test-slave-kk5c"
server_key: "f627e81bf6cd720e577c675fcd0bc5c2"
server_key_iv: "5adf97776ae619104330d5198d123673"
server_key_version: "encryptionkey@0"
I20260620 21:50:17.825639 2102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27"
format_stamp: "Formatted at 2026-06-20 21:50:17 on dist-test-slave-kk5c"
server_key: "f627e81bf6cd720e577c675fcd0bc5c2"
server_key_iv: "5adf97776ae619104330d5198d123673"
server_key_version: "encryptionkey@0"
I20260620 21:50:17.829747 2102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:17.832614 2117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:17.834014 2102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:50:17.834175 2102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27"
format_stamp: "Formatted at 2026-06-20 21:50:17 on dist-test-slave-kk5c"
server_key: "f627e81bf6cd720e577c675fcd0bc5c2"
server_key_iv: "5adf97776ae619104330d5198d123673"
server_key_version: "encryptionkey@0"
I20260620 21:50:17.834322 2102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:17.855849 2102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:17.859448 2102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:17.859666 2102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:17.861116 2102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:17.861191 2102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:17.861265 2102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:17.861305 2102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:17.871668 2102 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:33391
I20260620 21:50:17.871788 2232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:33391 every 8 connection(s)
Jun 20 21:50:17 dist-test-slave-kk5c krb5kdc[1845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:17.872820 2102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:17.877595 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:17.862094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:34371 (local address 127.0.193.254:34783)
0620 21:50:17.862240 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:17.862244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:17.863383 (+ 1139us) server_negotiation.cc:407] Connection header received
0620 21:50:17.864557 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:17.864561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:17.864639 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:17.864755 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:17.866413 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.867377 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.869040 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.869419 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.869549 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:17.872764 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:17.872767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:17.872769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:17.872796 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:17.874580 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:17.875662 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:17.875666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:17.875667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:17.875776 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:17.876256 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:17.876259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:17.876260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:17.876549 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:17.876671 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:17.877305 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 21:50:17.877454 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:50:17.878535 2126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:17.862677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:34783 (local address 127.0.193.194:34371)
0620 21:50:17.863171 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:17.863280 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:17.864306 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:17.864956 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:17.864967 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:17.865447 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:17.866212 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:17.866233 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.867544 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.867547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:17.868905 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:17.868915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.869600 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.869605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:17.869834 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:17.870561 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:17.870591 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:17.872510 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:17.874791 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:17.874796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:17.874808 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:17.875489 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:17.875900 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:17.875902 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:17.875904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:17.876098 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:17.876717 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:17.876723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:17.876998 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 21:50:17.877232 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 21:50:17.877564 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":138,"threads_started":1}
I20260620 21:50:17.878568 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 2102
I20260620 21:50:17.878670 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:50:17.878943 775 external_mini_cluster.cc:1428] Setting key dc0dc231dce758247d564d75e721efe8
I20260620 21:50:17.880172 2233 heartbeater.cc:347] Connected to a master server at 127.0.193.254:34783
I20260620 21:50:17.880378 2233 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:17.881023 2233 heartbeater.cc:511] Master 127.0.193.254:34783 requested a full tablet report, sending...
I20260620 21:50:17.882249 1889 ts_manager.cc:194] Registered new tserver with Master: 120f6d32a66c4ee4a0aa3c35cfce1e27 (127.0.193.194:33391)
I20260620 21:50:17.883052 1889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:34371
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
I20260620 21:50:17.891832 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:17.882770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:58025 (local address 127.0.193.254:34783)
0620 21:50:17.882873 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:17.882877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:17.883016 (+ 139us) server_negotiation.cc:407] Connection header received
0620 21:50:17.883151 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:17.883155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:17.883236 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:17.883332 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:17.884006 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.884894 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.886322 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.886821 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.886928 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:17.888481 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:17.888484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:17.888486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:17.888513 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:17.890045 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:17.890557 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:17.890559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:17.890561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:17.890640 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:17.890890 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:17.890893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:17.890894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:17.891112 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:17.891263 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:17.891582 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 21:50:17.891688 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 21:50:17.891837 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:17.882412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:34783 (local address 127.0.193.194:58025)
0620 21:50:17.882931 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:17.882945 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:17.883055 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:17.883441 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:17.883444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:17.883683 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:17.883873 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:17.883886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.885047 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.885053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:17.886165 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:17.886180 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:17.886963 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:17.886970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:17.887117 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:17.887700 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:17.887716 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:17.888297 (+ 581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:17.890196 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:17.890201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:17.890203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:17.890468 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:17.890729 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:17.890731 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:17.890733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:17.890814 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:17.891305 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:17.891309 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:17.891423 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 21:50:17.891528 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 21:50:17.891668 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":43,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:50:17.911674 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:34783
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:18.028936 2242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:18.029229 2242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:18.029325 2242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:18.033593 2242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:18.033679 2242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:18.033720 2242 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:50:18.033795 2242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:18.039898 2242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.193.254:34783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:18.041114 2242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:18.042346 2242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:18.052703 2247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:18.052698 2250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:18.052748 2242 server_base.cc:1034] running on GCE node
W20260620 21:50:18.052698 2248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:18.053498 2242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:18.054279 2242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:18.055469 2242 hybrid_clock.cc:648] HybridClock initialized: now 1781992218055455 us; error 29 us; skew 500 ppm
Jun 20 21:50:18 dist-test-slave-kk5c krb5kdc[1845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:18.058430 2242 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:50:18.059837 2242 webserver.cc:460] Webserver started at http://127.0.193.195:46051/ using document root <none> and password file <none>
I20260620 21:50:18.060432 2242 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:18.060494 2242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:18.060715 2242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:18.062820 2242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "a2481c04e9c74e0c92cb715c8ac2d15f"
format_stamp: "Formatted at 2026-06-20 21:50:18 on dist-test-slave-kk5c"
server_key: "deca575e6c3471de4ae831f773681776"
server_key_iv: "eda602c67d5fdbfa4eb869df79f63524"
server_key_version: "encryptionkey@0"
I20260620 21:50:18.063403 2242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "a2481c04e9c74e0c92cb715c8ac2d15f"
format_stamp: "Formatted at 2026-06-20 21:50:18 on dist-test-slave-kk5c"
server_key: "deca575e6c3471de4ae831f773681776"
server_key_iv: "eda602c67d5fdbfa4eb869df79f63524"
server_key_version: "encryptionkey@0"
I20260620 21:50:18.067325 2242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:50:18.070081 2257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:18.071910 2242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:18.072129 2242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "a2481c04e9c74e0c92cb715c8ac2d15f"
format_stamp: "Formatted at 2026-06-20 21:50:18 on dist-test-slave-kk5c"
server_key: "deca575e6c3471de4ae831f773681776"
server_key_iv: "eda602c67d5fdbfa4eb869df79f63524"
server_key_version: "encryptionkey@0"
I20260620 21:50:18.072310 2242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:18.091159 2242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:18.094748 2242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:18.094961 2242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:18.096839 2242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:18.096942 2242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:18.097011 2242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:18.097052 2242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:50:18 dist-test-slave-kk5c krb5kdc[1845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:18.109193 2242 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:43191
I20260620 21:50:18.109274 2372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:43191 every 8 connection(s)
I20260620 21:50:18.110422 2242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:50:18.113013 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.097848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:60005 (local address 127.0.193.254:34783)
0620 21:50:18.097979 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.097983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.099459 (+ 1476us) server_negotiation.cc:407] Connection header received
0620 21:50:18.100657 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.100663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.100729 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.100793 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:18.102632 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.103697 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.105629 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.106001 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.106179 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.109079 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:18.109082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:18.109084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:18.109111 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:18.110840 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:18.111505 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:18.111509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:18.111519 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:18.111609 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:18.111939 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:18.111942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:18.111944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:18.112143 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:18.112244 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.112720 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.112842 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:50:18.113785 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.098654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:34783 (local address 127.0.193.195:60005)
0620 21:50:18.099304 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:18.099351 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:18.100379 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:18.101257 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:18.101268 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:18.101733 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:18.102440 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.102461 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.103866 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.103873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.105453 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.105470 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.106215 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.106219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.106382 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.107285 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:18.107302 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:18.108809 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:18.111004 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:18.111019 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:18.111034 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:18.111400 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:18.111702 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:18.111705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:18.111707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:18.111839 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:18.112240 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:18.112246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:18.112488 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 21:50:18.112664 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 21:50:18.112978 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":156,"threads_started":1}
I20260620 21:50:18.115041 2373 heartbeater.cc:347] Connected to a master server at 127.0.193.254:34783
I20260620 21:50:18.115252 2373 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:18.115739 2373 heartbeater.cc:511] Master 127.0.193.254:34783 requested a full tablet report, sending...
I20260620 21:50:18.116838 1890 ts_manager.cc:194] Registered new tserver with Master: a2481c04e9c74e0c92cb715c8ac2d15f (127.0.193.195:43191)
I20260620 21:50:18.117641 1890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:60005
I20260620 21:50:18.120540 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 2242
I20260620 21:50:18.120723 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:50:18.121060 775 external_mini_cluster.cc:1428] Setting key f4e07d74461e5bf460c21bdd59423d5c
I20260620 21:50:18.123603 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:50:18.125233 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.117418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:49633 (local address 127.0.193.254:34783)
0620 21:50:18.117542 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.117547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.117805 (+ 258us) server_negotiation.cc:407] Connection header received
0620 21:50:18.117961 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.117964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.118003 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.118091 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:18.118518 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.119024 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.119905 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.120137 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.120278 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.121658 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:18.121662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:18.121665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:18.121699 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:18.123601 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:18.124050 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:18.124055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:18.124056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:18.124145 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:18.124404 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:18.124410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:18.124412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:18.124606 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:18.124707 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.124970 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.125081 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:50:18.125247 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.117299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:34783 (local address 127.0.193.195:49633)
0620 21:50:18.117710 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:18.117736 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:18.117872 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:18.118106 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:18.118109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:18.118311 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:18.118406 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.118412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.119148 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.119152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.119784 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.119793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.120247 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.120252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.120339 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.121075 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:18.121093 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:18.121507 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:18.123715 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:18.123719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:18.123721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:18.123952 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:18.124231 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:18.124235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:18.124237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:18.124318 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:18.124706 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:18.124710 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:18.124806 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 21:50:18.124905 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 21:50:18.125061 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":115,"threads_started":1}
Jun 20 21:50:18 dist-test-slave-kk5c krb5kdc[1845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992218, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:50:18 dist-test-slave-kk5c krb5kdc[1845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992218, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:18.151328 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.142493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.0.193.254:34783)
0620 21:50:18.142667 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.142670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.142685 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:50:18.142727 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.142730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.142769 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.142846 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:18.143716 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.144263 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.145131 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.145346 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.145426 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.148169 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:18.148172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:18.148175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:18.148205 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:18.149747 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:18.150171 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:18.150174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:18.150175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:18.150248 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:18.150492 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:18.150493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:18.150495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:18.150706 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:18.150806 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.151016 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.151111 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 21:50:18.154408 1890 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:50:18.157689 1890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:18.172542 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.166419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:43191 (local address 127.0.0.1:36372)
0620 21:50:18.166858 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:18.166915 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:18.167062 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:18.167996 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:18.168002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:18.168024 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:18.168182 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.168189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.169364 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.169371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.170266 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.170276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.171888 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.171892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.172037 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.172107 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 21:50:18.172238 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 21:50:18.172381 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"spinlock_wait_cycles":3200,"thread_start_us":83,"threads_started":1}
I20260620 21:50:18.173228 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.166519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.0.193.195:43191)
0620 21:50:18.167429 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.167442 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.167469 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:50:18.167559 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.167567 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.167802 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.168004 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:18.168295 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.169268 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.170397 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.171734 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.171855 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.172043 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.172942 (+ 899us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.173051 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":135,"threads_started":1}
I20260620 21:50:18.173722 2395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.167472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.0.193.194:33391)
0620 21:50:18.167863 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.167876 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.167899 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:50:18.168620 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.168627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.168816 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.168949 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:18.169191 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.170426 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.172123 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.172842 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.172946 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.173077 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.173484 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.173591 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":109,"threads_started":1}
I20260620 21:50:18.173813 2390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.166474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:38369 (local address 127.0.0.1:37228)
0620 21:50:18.166931 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:18.166947 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:18.167062 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:18.168509 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:18.168514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:18.168534 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:18.168703 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.168711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.170516 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.170519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.172129 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.172139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.173186 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.173189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.173520 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.173539 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 21:50:18.173606 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 21:50:18.173660 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":214,"spinlock_wait_cycles":6144,"thread_start_us":107,"threads_started":1}
I20260620 21:50:18.173676 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.167392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:33391 (local address 127.0.0.1:46302)
0620 21:50:18.167730 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:18.167744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:18.168462 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:18.168956 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:18.168959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:18.168977 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:18.169066 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.169072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.170553 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.170557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.171963 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.171972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.173154 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.173159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.173332 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.173363 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 21:50:18.173436 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 21:50:18.173519 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":240,"thread_start_us":105,"threads_started":1}
I20260620 21:50:18.174312 2394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.167397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.0.193.193:38369)
0620 21:50:18.167992 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.167998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.168019 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:50:18.168093 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.168097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.168354 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.168849 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:18.168993 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.170228 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.172261 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.173062 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.173175 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.173315 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.173751 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.174185 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":96,"threads_started":1}
I20260620 21:50:18.176532 2163 tablet_service.cc:1467] Processing CreateTablet for tablet bcefd2804cd243c9a891074a3e3b1471 (DEFAULT_TABLE table=test-table [id=aff06ccf624d4ec192c865b509b5126e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:18.176529 2307 tablet_service.cc:1467] Processing CreateTablet for tablet bcefd2804cd243c9a891074a3e3b1471 (DEFAULT_TABLE table=test-table [id=aff06ccf624d4ec192c865b509b5126e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:18.177119 2027 tablet_service.cc:1467] Processing CreateTablet for tablet bcefd2804cd243c9a891074a3e3b1471 (DEFAULT_TABLE table=test-table [id=aff06ccf624d4ec192c865b509b5126e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:18.177647 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcefd2804cd243c9a891074a3e3b1471. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:18.177654 2307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcefd2804cd243c9a891074a3e3b1471. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:18.178022 2027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcefd2804cd243c9a891074a3e3b1471. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:18.185288 2396 tablet_bootstrap.cc:492] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345: Bootstrap starting.
I20260620 21:50:18.185783 2397 tablet_bootstrap.cc:492] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f: Bootstrap starting.
I20260620 21:50:18.186409 2398 tablet_bootstrap.cc:492] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Bootstrap starting.
I20260620 21:50:18.188009 2397 tablet_bootstrap.cc:654] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:18.188594 2396 tablet_bootstrap.cc:654] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:18.188593 2398 tablet_bootstrap.cc:654] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:18.188812 2397 log.cc:824] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:18.189679 2398 log.cc:824] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:18.189679 2396 log.cc:824] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:18.191774 2397 tablet_bootstrap.cc:492] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f: No bootstrap required, opened a new log
I20260620 21:50:18.191817 2398 tablet_bootstrap.cc:492] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27: No bootstrap required, opened a new log
I20260620 21:50:18.191810 2396 tablet_bootstrap.cc:492] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345: No bootstrap required, opened a new log
I20260620 21:50:18.192008 2397 ts_tablet_manager.cc:1397] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:50:18.192014 2398 ts_tablet_manager.cc:1397] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 21:50:18.192013 2396 ts_tablet_manager.cc:1397] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 21:50:18.195545 2397 raft_consensus.cc:348] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } }
I20260620 21:50:18.195839 2397 raft_consensus.cc:374] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:18.195894 2397 raft_consensus.cc:729] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2481c04e9c74e0c92cb715c8ac2d15f, State: Initialized, Role: FOLLOWER
I20260620 21:50:18.196365 2397 consensus_queue.cc:260] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } }
I20260620 21:50:18.196785 2398 raft_consensus.cc:348] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } }
I20260620 21:50:18.197070 2398 raft_consensus.cc:374] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:18.197129 2398 raft_consensus.cc:729] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120f6d32a66c4ee4a0aa3c35cfce1e27, State: Initialized, Role: FOLLOWER
I20260620 21:50:18.197070 2396 raft_consensus.cc:348] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } }
I20260620 21:50:18.197384 2396 raft_consensus.cc:374] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:18.197448 2396 raft_consensus.cc:729] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb15b8c9da5a41198e80a18045f24345, State: Initialized, Role: FOLLOWER
I20260620 21:50:18.197609 2398 consensus_queue.cc:260] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } }
I20260620 21:50:18.197858 2397 ts_tablet_manager.cc:1428] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:50:18.197857 2396 consensus_queue.cc:260] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } }
I20260620 21:50:18.198254 2373 heartbeater.cc:503] Master 127.0.193.254:34783 was elected leader, sending a full tablet report...
I20260620 21:50:18.198362 2233 heartbeater.cc:503] Master 127.0.193.254:34783 was elected leader, sending a full tablet report...
I20260620 21:50:18.198629 2398 ts_tablet_manager.cc:1428] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 21:50:18.199615 2093 heartbeater.cc:503] Master 127.0.193.254:34783 was elected leader, sending a full tablet report...
I20260620 21:50:18.199954 2396 ts_tablet_manager.cc:1428] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260620 21:50:18.208180 2404 raft_consensus.cc:482] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:18.208412 2404 raft_consensus.cc:504] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } }
I20260620 21:50:18.209550 2404 leader_election.cc:288] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2481c04e9c74e0c92cb715c8ac2d15f (127.0.193.195:43191), 120f6d32a66c4ee4a0aa3c35cfce1e27 (127.0.193.194:33391)
I20260620 21:50:18.213745 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.209813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:43191 (local address 127.0.193.193:52071)
0620 21:50:18.210190 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:18.210204 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:18.210351 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:18.210668 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:18.210671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:18.210689 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:18.210811 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.210823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.211605 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.211610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.212478 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.212487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.213129 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.213132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.213318 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.213346 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:50:18.213462 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 21:50:18.213563 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":39,"thread_start_us":179,"threads_started":1}
I20260620 21:50:18.213757 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.209942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:52071 (local address 127.0.193.195:43191)
0620 21:50:18.210149 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.210154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.210313 (+ 159us) server_negotiation.cc:407] Connection header received
0620 21:50:18.210460 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.210468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.210532 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.210603 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:18.210923 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.211497 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.212599 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.213005 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.213131 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.213261 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.213480 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.213563 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260620 21:50:18.213925 2395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.210451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:60745 (local address 127.0.193.194:33391)
0620 21:50:18.210603 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.210608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.210767 (+ 159us) server_negotiation.cc:407] Connection header received
0620 21:50:18.210906 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.210909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.210950 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.211026 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:18.211316 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.211827 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.212943 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.213346 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.213478 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.213542 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.213728 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.213800 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:50:18.213939 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.210356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:33391 (local address 127.0.193.193:60745)
0620 21:50:18.210680 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:18.210692 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:18.210806 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:18.211051 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:18.211053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:18.211068 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:18.211177 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.211230 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.211935 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.211941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.212823 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.212832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.213468 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.213472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.213607 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.213631 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:50:18.213715 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:50:18.213781 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":23,"thread_start_us":106,"threads_started":1}
I20260620 21:50:18.214799 2187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcefd2804cd243c9a891074a3e3b1471" candidate_uuid: "cb15b8c9da5a41198e80a18045f24345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" is_pre_election: true
I20260620 21:50:18.214798 2327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcefd2804cd243c9a891074a3e3b1471" candidate_uuid: "cb15b8c9da5a41198e80a18045f24345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" is_pre_election: true
I20260620 21:50:18.215127 2187 raft_consensus.cc:2454] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb15b8c9da5a41198e80a18045f24345 in term 0.
I20260620 21:50:18.215127 2327 raft_consensus.cc:2454] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb15b8c9da5a41198e80a18045f24345 in term 0.
I20260620 21:50:18.215662 1980 leader_election.cc:302] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 120f6d32a66c4ee4a0aa3c35cfce1e27, cb15b8c9da5a41198e80a18045f24345; no voters:
I20260620 21:50:18.215957 2404 raft_consensus.cc:2781] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:18.216063 2404 raft_consensus.cc:482] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:18.216117 2404 raft_consensus.cc:3037] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:18.217414 2404 raft_consensus.cc:504] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } }
I20260620 21:50:18.217775 2404 leader_election.cc:288] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [CANDIDATE]: Term 1 election: Requested vote from peers a2481c04e9c74e0c92cb715c8ac2d15f (127.0.193.195:43191), 120f6d32a66c4ee4a0aa3c35cfce1e27 (127.0.193.194:33391)
I20260620 21:50:18.218163 2327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcefd2804cd243c9a891074a3e3b1471" candidate_uuid: "cb15b8c9da5a41198e80a18045f24345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f"
I20260620 21:50:18.218207 2187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcefd2804cd243c9a891074a3e3b1471" candidate_uuid: "cb15b8c9da5a41198e80a18045f24345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27"
I20260620 21:50:18.218312 2327 raft_consensus.cc:3037] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:18.218334 2187 raft_consensus.cc:3037] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:18.219477 2327 raft_consensus.cc:2454] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb15b8c9da5a41198e80a18045f24345 in term 1.
I20260620 21:50:18.219493 2187 raft_consensus.cc:2454] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb15b8c9da5a41198e80a18045f24345 in term 1.
I20260620 21:50:18.219799 1980 leader_election.cc:302] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2481c04e9c74e0c92cb715c8ac2d15f, cb15b8c9da5a41198e80a18045f24345; no voters:
I20260620 21:50:18.220001 2404 raft_consensus.cc:2781] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:18.220263 2404 raft_consensus.cc:686] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [term 1 LEADER]: Becoming Leader. State: Replica: cb15b8c9da5a41198e80a18045f24345, State: Running, Role: LEADER
I20260620 21:50:18.220608 2404 consensus_queue.cc:237] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } }
I20260620 21:50:18.224225 1888 catalog_manager.cc:5495] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb15b8c9da5a41198e80a18045f24345 (127.0.193.193). New cstate: current_term: 1 leader_uuid: "cb15b8c9da5a41198e80a18045f24345" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } health_report { overall_health: UNKNOWN } } }
I20260620 21:50:18.244608 2394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.240637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.0.193.193:38369)
0620 21:50:18.240761 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.240765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.240847 (+ 82us) server_negotiation.cc:407] Connection header received
0620 21:50:18.241125 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.241128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.241189 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.241291 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:18.241631 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.242151 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.243115 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.243389 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.243466 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.243829 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:18.244271 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:18.244359 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.244437 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.244487 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 21:50:18.250775 2327 raft_consensus.cc:1261] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f [term 1 FOLLOWER]: Refusing update from remote peer cb15b8c9da5a41198e80a18045f24345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:18.250795 2187 raft_consensus.cc:1261] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 1 FOLLOWER]: Refusing update from remote peer cb15b8c9da5a41198e80a18045f24345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:18.251464 2404 consensus_queue.cc:1035] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:18.251705 2407 consensus_queue.cc:1035] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:18.259181 2410 mvcc.cc:205] Tried to move back new op lower bound from 7299040125946978304 to 7299040125834354688. Current Snapshot: MvccSnapshot[applied={T|T < 7299040125946978304}]
I20260620 21:50:18.273152 1888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:18.273330 1888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:18.275957 1888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0: Sending DeleteTablet for 3 replicas of tablet bcefd2804cd243c9a891074a3e3b1471
I20260620 21:50:18.276892 2027 tablet_service.cc:1514] Processing DeleteTablet for tablet bcefd2804cd243c9a891074a3e3b1471 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:18 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:37228
I20260620 21:50:18.276933 2163 tablet_service.cc:1514] Processing DeleteTablet for tablet bcefd2804cd243c9a891074a3e3b1471 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:18 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:46302
I20260620 21:50:18.277544 2420 tablet_replica.cc:330] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345: stopping tablet replica
I20260620 21:50:18.277650 2307 tablet_service.cc:1514] Processing DeleteTablet for tablet bcefd2804cd243c9a891074a3e3b1471 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:18 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:36372
I20260620 21:50:18.278054 2420 raft_consensus.cc:2229] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:50:18.278198 2419 tablet_replica.cc:330] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27: stopping tablet replica
I20260620 21:50:18.278335 1888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 21:50:18.278502 2419 raft_consensus.cc:2229] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:18.278736 2420 raft_consensus.cc:2258] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:18.278813 2419 raft_consensus.cc:2258] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 21:50:18.278820 1888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:18.279726 2421 tablet_replica.cc:330] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f: stopping tablet replica
I20260620 21:50:18.280102 2421 raft_consensus.cc:2229] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:18.280416 2421 raft_consensus.cc:2258] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:18.281124 2420 ts_tablet_manager.cc:1905] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:18.281672 2419 ts_tablet_manager.cc:1905] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:18.281908 2421 ts_tablet_manager.cc:1905] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:18.284106 2027 tablet_service.cc:1467] Processing CreateTablet for tablet a08c6c5cb68849159db6ad0c21ef4bd0 (DEFAULT_TABLE table=test-table [id=009cbdda8b104ebb8bbd0f2d2b332d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:18.284346 2307 tablet_service.cc:1467] Processing CreateTablet for tablet a08c6c5cb68849159db6ad0c21ef4bd0 (DEFAULT_TABLE table=test-table [id=009cbdda8b104ebb8bbd0f2d2b332d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:18.284485 2027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08c6c5cb68849159db6ad0c21ef4bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:18.284917 2163 tablet_service.cc:1467] Processing CreateTablet for tablet a08c6c5cb68849159db6ad0c21ef4bd0 (DEFAULT_TABLE table=test-table [id=009cbdda8b104ebb8bbd0f2d2b332d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:18.285178 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08c6c5cb68849159db6ad0c21ef4bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:18.286906 2396 tablet_bootstrap.cc:492] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345: Bootstrap starting.
I20260620 21:50:18.287434 2307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08c6c5cb68849159db6ad0c21ef4bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:18.287518 2398 tablet_bootstrap.cc:492] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Bootstrap starting.
I20260620 21:50:18.287968 2396 tablet_bootstrap.cc:654] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:18.288453 2398 tablet_bootstrap.cc:654] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:18.288875 2419 ts_tablet_manager.cc:1918] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:18.289031 2420 ts_tablet_manager.cc:1918] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:18.289150 2420 log.cc:1196] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/wal/wals/bcefd2804cd243c9a891074a3e3b1471
I20260620 21:50:18.289499 2420 ts_tablet_manager.cc:1939] T bcefd2804cd243c9a891074a3e3b1471 P cb15b8c9da5a41198e80a18045f24345: Deleting consensus metadata
I20260620 21:50:18.290544 2397 tablet_bootstrap.cc:492] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f: Bootstrap starting.
I20260620 21:50:18.290602 2419 log.cc:1196] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/bcefd2804cd243c9a891074a3e3b1471
I20260620 21:50:18.290714 1875 catalog_manager.cc:4841] TS cb15b8c9da5a41198e80a18045f24345 (127.0.193.193:38369): tablet bcefd2804cd243c9a891074a3e3b1471 (table test-table [id=aff06ccf624d4ec192c865b509b5126e]) successfully deleted
I20260620 21:50:18.290918 2419 ts_tablet_manager.cc:1939] T bcefd2804cd243c9a891074a3e3b1471 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Deleting consensus metadata
I20260620 21:50:18.291611 2397 tablet_bootstrap.cc:654] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:18.292151 1876 catalog_manager.cc:4841] TS 120f6d32a66c4ee4a0aa3c35cfce1e27 (127.0.193.194:33391): tablet bcefd2804cd243c9a891074a3e3b1471 (table test-table [id=aff06ccf624d4ec192c865b509b5126e]) successfully deleted
I20260620 21:50:18.292580 2421 ts_tablet_manager.cc:1918] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:18.292672 2421 log.cc:1196] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/bcefd2804cd243c9a891074a3e3b1471
I20260620 21:50:18.293009 2421 ts_tablet_manager.cc:1939] T bcefd2804cd243c9a891074a3e3b1471 P a2481c04e9c74e0c92cb715c8ac2d15f: Deleting consensus metadata
I20260620 21:50:18.293089 2398 tablet_bootstrap.cc:492] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27: No bootstrap required, opened a new log
I20260620 21:50:18.293181 2398 ts_tablet_manager.cc:1397] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260620 21:50:18.293628 2398 raft_consensus.cc:348] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.293745 2398 raft_consensus.cc:374] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:18.293785 2398 raft_consensus.cc:729] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120f6d32a66c4ee4a0aa3c35cfce1e27, State: Initialized, Role: FOLLOWER
I20260620 21:50:18.293905 2398 consensus_queue.cc:260] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.294174 2398 ts_tablet_manager.cc:1428] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:18.294903 1876 catalog_manager.cc:4841] TS a2481c04e9c74e0c92cb715c8ac2d15f (127.0.193.195:43191): tablet bcefd2804cd243c9a891074a3e3b1471 (table test-table [id=aff06ccf624d4ec192c865b509b5126e]) successfully deleted
I20260620 21:50:18.295598 2396 tablet_bootstrap.cc:492] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345: No bootstrap required, opened a new log
I20260620 21:50:18.295696 2396 ts_tablet_manager.cc:1397] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260620 21:50:18.295840 2397 tablet_bootstrap.cc:492] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f: No bootstrap required, opened a new log
I20260620 21:50:18.295920 2397 ts_tablet_manager.cc:1397] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260620 21:50:18.296232 2396 raft_consensus.cc:348] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.296355 2396 raft_consensus.cc:374] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:18.296389 2396 raft_consensus.cc:729] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb15b8c9da5a41198e80a18045f24345, State: Initialized, Role: FOLLOWER
I20260620 21:50:18.296347 2397 raft_consensus.cc:348] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.296447 2397 raft_consensus.cc:374] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:18.296478 2397 raft_consensus.cc:729] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2481c04e9c74e0c92cb715c8ac2d15f, State: Initialized, Role: FOLLOWER
I20260620 21:50:18.296523 2396 consensus_queue.cc:260] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.296608 2397 consensus_queue.cc:260] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.296788 2396 ts_tablet_manager.cc:1428] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:50:18.296895 2397 ts_tablet_manager.cc:1428] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260620 21:50:18.361501 2374 tablet.cc:2369] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:50:18.374262 2234 tablet.cc:2369] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:50:18.386814 2094 tablet.cc:2369] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:50:18.456305 2403 raft_consensus.cc:482] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:18.456470 2403 raft_consensus.cc:504] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.457641 2403 leader_election.cc:288] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb15b8c9da5a41198e80a18045f24345 (127.0.193.193:38369), a2481c04e9c74e0c92cb715c8ac2d15f (127.0.193.195:43191)
I20260620 21:50:18.461621 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.458054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:52889 (local address 127.0.193.195:43191)
0620 21:50:18.458282 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.458288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.458460 (+ 172us) server_negotiation.cc:407] Connection header received
0620 21:50:18.458642 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.458644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.458709 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.458785 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:18.459053 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.459590 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.460646 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.461037 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.461134 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.461176 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.461389 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.461471 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260620 21:50:18.461642 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.457963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:43191 (local address 127.0.193.194:52889)
0620 21:50:18.458353 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:18.458368 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:18.458469 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:18.458809 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:18.458812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:18.458825 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:18.458927 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.458931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.459691 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.459697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.460530 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.460538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.461145 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.461148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.461254 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.461280 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:50:18.461365 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 21:50:18.461451 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":86,"thread_start_us":73,"threads_started":1}
I20260620 21:50:18.461724 2394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.458148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:45279 (local address 127.0.193.193:38369)
0620 21:50:18.458325 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.458330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.458356 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:50:18.458552 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.458555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.458621 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.458677 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:18.458998 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.459625 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.460695 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.461157 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.461270 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.461323 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.461521 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.461600 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 21:50:18.462141 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.457963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:38369 (local address 127.0.193.194:45279)
0620 21:50:18.458280 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:18.458295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:18.458413 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:18.458708 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:18.458712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:18.458730 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:18.458882 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.458889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.459758 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.459761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.460563 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.460572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.461276 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.461278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.461373 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.461414 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 21:50:18.461497 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 21:50:18.461560 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260620 21:50:18.462106 2327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08c6c5cb68849159db6ad0c21ef4bd0" candidate_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" is_pre_election: true
I20260620 21:50:18.462251 2327 raft_consensus.cc:2454] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 120f6d32a66c4ee4a0aa3c35cfce1e27 in term 0.
I20260620 21:50:18.462610 2120 leader_election.cc:302] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 120f6d32a66c4ee4a0aa3c35cfce1e27, a2481c04e9c74e0c92cb715c8ac2d15f; no voters:
I20260620 21:50:18.462774 2047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08c6c5cb68849159db6ad0c21ef4bd0" candidate_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb15b8c9da5a41198e80a18045f24345" is_pre_election: true
I20260620 21:50:18.462881 2403 raft_consensus.cc:2781] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:18.462930 2403 raft_consensus.cc:482] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:18.462976 2403 raft_consensus.cc:3037] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:18.463014 2047 raft_consensus.cc:2454] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 120f6d32a66c4ee4a0aa3c35cfce1e27 in term 0.
I20260620 21:50:18.464123 2403 raft_consensus.cc:504] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.464483 2403 leader_election.cc:288] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [CANDIDATE]: Term 1 election: Requested vote from peers cb15b8c9da5a41198e80a18045f24345 (127.0.193.193:38369), a2481c04e9c74e0c92cb715c8ac2d15f (127.0.193.195:43191)
I20260620 21:50:18.464864 2047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08c6c5cb68849159db6ad0c21ef4bd0" candidate_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb15b8c9da5a41198e80a18045f24345"
I20260620 21:50:18.464893 2327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08c6c5cb68849159db6ad0c21ef4bd0" candidate_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f"
I20260620 21:50:18.465008 2047 raft_consensus.cc:3037] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:18.465008 2327 raft_consensus.cc:3037] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:18.466142 2327 raft_consensus.cc:2454] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 120f6d32a66c4ee4a0aa3c35cfce1e27 in term 1.
I20260620 21:50:18.466173 2047 raft_consensus.cc:2454] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 120f6d32a66c4ee4a0aa3c35cfce1e27 in term 1.
I20260620 21:50:18.466495 2120 leader_election.cc:302] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 120f6d32a66c4ee4a0aa3c35cfce1e27, a2481c04e9c74e0c92cb715c8ac2d15f, cb15b8c9da5a41198e80a18045f24345; no voters:
I20260620 21:50:18.466663 2403 raft_consensus.cc:2781] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:18.466871 2403 raft_consensus.cc:686] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 1 LEADER]: Becoming Leader. State: Replica: 120f6d32a66c4ee4a0aa3c35cfce1e27, State: Running, Role: LEADER
I20260620 21:50:18.467141 2403 consensus_queue.cc:237] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.470218 1889 catalog_manager.cc:5495] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 120f6d32a66c4ee4a0aa3c35cfce1e27 (127.0.193.194). New cstate: current_term: 1 leader_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } health_report { overall_health: UNKNOWN } } }
Jun 20 21:50:18 dist-test-slave-kk5c krb5kdc[1845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:18.526996 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.517100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:34783 (local address 127.0.0.1:42990)
0620 21:50:18.517404 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:18.517417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:18.517521 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:18.517757 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:18.517759 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:18.518164 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:18.518263 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.518269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.519111 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.519114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.519795 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.519805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.520338 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.520341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.520410 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.521025 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:18.521038 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:18.522967 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:18.525122 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:18.525133 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:18.525139 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:18.525410 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:18.525811 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:18.525814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:18.525816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:18.525923 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:18.526414 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:18.526420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:18.526549 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 21:50:18.526657 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 21:50:18.526802 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":128,"threads_started":1}
I20260620 21:50:18.526998 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.517162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.0.193.254:34783)
0620 21:50:18.517298 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.517303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.517489 (+ 186us) server_negotiation.cc:407] Connection header received
0620 21:50:18.517604 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.517607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.517647 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.517720 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:18.518395 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.518962 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.519982 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.520206 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.520291 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.523144 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:18.523148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:18.523150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:18.523181 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:18.524903 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:18.525585 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:18.525588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:18.525589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:18.525687 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:18.526042 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:18.526045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:18.526046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:18.526290 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:18.526381 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.526750 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.526866 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:50:18.531628 1889 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:42990:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 21:50:18.532397 1889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:18.537843 2027 tablet_service.cc:1467] Processing CreateTablet for tablet 091d0379c1dc4b5bab0e49014653e289 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b777bd2b15ef429ea0612d2d269c8fe3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:50:18.537876 2163 tablet_service.cc:1467] Processing CreateTablet for tablet 091d0379c1dc4b5bab0e49014653e289 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b777bd2b15ef429ea0612d2d269c8fe3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:50:18.538318 2027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 091d0379c1dc4b5bab0e49014653e289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:18.538321 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 091d0379c1dc4b5bab0e49014653e289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:18.539711 2307 tablet_service.cc:1467] Processing CreateTablet for tablet 091d0379c1dc4b5bab0e49014653e289 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b777bd2b15ef429ea0612d2d269c8fe3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:50:18.540120 2307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 091d0379c1dc4b5bab0e49014653e289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:18.541339 2396 tablet_bootstrap.cc:492] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345: Bootstrap starting.
I20260620 21:50:18.541821 2398 tablet_bootstrap.cc:492] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Bootstrap starting.
I20260620 21:50:18.542280 2396 tablet_bootstrap.cc:654] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:18.542780 2398 tablet_bootstrap.cc:654] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:18.543121 2397 tablet_bootstrap.cc:492] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f: Bootstrap starting.
I20260620 21:50:18.544167 2397 tablet_bootstrap.cc:654] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:18.544445 2398 tablet_bootstrap.cc:492] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27: No bootstrap required, opened a new log
I20260620 21:50:18.544530 2398 ts_tablet_manager.cc:1397] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:50:18.545145 2396 tablet_bootstrap.cc:492] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345: No bootstrap required, opened a new log
I20260620 21:50:18.545073 2398 raft_consensus.cc:348] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.545217 2398 raft_consensus.cc:374] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:18.545259 2396 ts_tablet_manager.cc:1397] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 21:50:18.545264 2398 raft_consensus.cc:729] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120f6d32a66c4ee4a0aa3c35cfce1e27, State: Initialized, Role: FOLLOWER
I20260620 21:50:18.545421 2397 tablet_bootstrap.cc:492] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f: No bootstrap required, opened a new log
I20260620 21:50:18.545420 2398 consensus_queue.cc:260] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.545512 2397 ts_tablet_manager.cc:1397] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:50:18.545706 2398 ts_tablet_manager.cc:1428] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 21:50:18.545727 2396 raft_consensus.cc:348] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.545833 2396 raft_consensus.cc:374] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:18.545869 2396 raft_consensus.cc:729] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb15b8c9da5a41198e80a18045f24345, State: Initialized, Role: FOLLOWER
I20260620 21:50:18.545919 2403 tablet_replica.cc:439] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } }
I20260620 21:50:18.545996 2397 raft_consensus.cc:348] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.546087 2403 tablet_replica.cc:442] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:18.545997 2396 consensus_queue.cc:260] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.546129 2397 raft_consensus.cc:374] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:18.546166 2397 raft_consensus.cc:729] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2481c04e9c74e0c92cb715c8ac2d15f, State: Initialized, Role: FOLLOWER
I20260620 21:50:18.546321 2396 ts_tablet_manager.cc:1428] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260620 21:50:18.546322 2397 consensus_queue.cc:260] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.546631 2397 ts_tablet_manager.cc:1428] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:50:18.546586 2402 tablet_replica.cc:439] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } }
I20260620 21:50:18.546715 2402 tablet_replica.cc:442] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:18.547024 2404 tablet_replica.cc:439] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } }
I20260620 21:50:18.547155 2404 tablet_replica.cc:442] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:18.849529 2403 consensus_queue.cc:1035] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:18.853919 2440 consensus_queue.cc:1035] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:18.855525 2440 raft_consensus.cc:482] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:18.856814 2440 raft_consensus.cc:504] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.857532 2440 leader_election.cc:288] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb15b8c9da5a41198e80a18045f24345 (127.0.193.193:38369), a2481c04e9c74e0c92cb715c8ac2d15f (127.0.193.195:43191)
I20260620 21:50:18.857873 2047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091d0379c1dc4b5bab0e49014653e289" candidate_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb15b8c9da5a41198e80a18045f24345" is_pre_election: true
I20260620 21:50:18.857973 2327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091d0379c1dc4b5bab0e49014653e289" candidate_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" is_pre_election: true
I20260620 21:50:18.858070 2047 raft_consensus.cc:2454] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 120f6d32a66c4ee4a0aa3c35cfce1e27 in term 0.
I20260620 21:50:18.858098 2327 raft_consensus.cc:2454] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 120f6d32a66c4ee4a0aa3c35cfce1e27 in term 0.
I20260620 21:50:18.858389 2119 leader_election.cc:302] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 120f6d32a66c4ee4a0aa3c35cfce1e27, cb15b8c9da5a41198e80a18045f24345; no voters:
I20260620 21:50:18.858561 2440 raft_consensus.cc:2781] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:18.858642 2440 raft_consensus.cc:482] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:18.858683 2440 raft_consensus.cc:3037] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:18.860016 2440 raft_consensus.cc:504] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.860388 2440 leader_election.cc:288] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [CANDIDATE]: Term 1 election: Requested vote from peers cb15b8c9da5a41198e80a18045f24345 (127.0.193.193:38369), a2481c04e9c74e0c92cb715c8ac2d15f (127.0.193.195:43191)
I20260620 21:50:18.860795 2047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091d0379c1dc4b5bab0e49014653e289" candidate_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb15b8c9da5a41198e80a18045f24345"
I20260620 21:50:18.860843 2327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091d0379c1dc4b5bab0e49014653e289" candidate_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f"
I20260620 21:50:18.860935 2047 raft_consensus.cc:3037] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:18.860955 2327 raft_consensus.cc:3037] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:18.862239 2047 raft_consensus.cc:2454] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 120f6d32a66c4ee4a0aa3c35cfce1e27 in term 1.
I20260620 21:50:18.862239 2327 raft_consensus.cc:2454] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 120f6d32a66c4ee4a0aa3c35cfce1e27 in term 1.
I20260620 21:50:18.862551 2120 leader_election.cc:302] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 120f6d32a66c4ee4a0aa3c35cfce1e27, a2481c04e9c74e0c92cb715c8ac2d15f; no voters:
I20260620 21:50:18.862768 2440 raft_consensus.cc:2781] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:18.862876 2440 raft_consensus.cc:686] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 1 LEADER]: Becoming Leader. State: Replica: 120f6d32a66c4ee4a0aa3c35cfce1e27, State: Running, Role: LEADER
I20260620 21:50:18.863008 2440 consensus_queue.cc:237] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } }
I20260620 21:50:18.863754 2427 tablet_replica.cc:439] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27: TxnStatusTablet state changed. Reason: New leader 120f6d32a66c4ee4a0aa3c35cfce1e27. Latest consensus state: current_term: 1 leader_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } }
I20260620 21:50:18.863855 2427 tablet_replica.cc:442] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:50:18.864136 2445 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 21:50:18.864353 2445 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 21:50:18.865253 1885 catalog_manager.cc:5495] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 120f6d32a66c4ee4a0aa3c35cfce1e27 (127.0.193.194). New cstate: current_term: 1 leader_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } health_report { overall_health: UNKNOWN } } }
I20260620 21:50:18.916800 2434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.911667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:33391 (local address 127.0.0.1:46318)
0620 21:50:18.911849 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:18.911869 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:18.912010 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:18.912390 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:18.912392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:18.912399 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:50:18.912506 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.912510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.914050 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.914053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.914791 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.914802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.915445 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.915449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.915581 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.915635 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:50:18.916490 (+ 855us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:50:18.916505 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 21:50:18.916565 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 21:50:18.916681 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 21:50:18.916836 2448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.911779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46318 (local address 127.0.193.194:33391)
0620 21:50:18.912078 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.912087 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.912102 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:50:18.912189 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.912192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.912255 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.912370 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:18.912608 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.913911 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.914912 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.915312 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.915410 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.915795 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:18.916381 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:18.916486 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.916589 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.916651 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":122,"threads_started":1}
I20260620 21:50:18.919965 2047 raft_consensus.cc:1261] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345 [term 1 FOLLOWER]: Refusing update from remote peer 120f6d32a66c4ee4a0aa3c35cfce1e27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:18.920091 2327 raft_consensus.cc:1261] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f [term 1 FOLLOWER]: Refusing update from remote peer 120f6d32a66c4ee4a0aa3c35cfce1e27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:18.920483 2440 consensus_queue.cc:1035] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:18.920674 2427 consensus_queue.cc:1035] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:18.922185 2404 tablet_replica.cc:439] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345: TxnStatusTablet state changed. Reason: New leader 120f6d32a66c4ee4a0aa3c35cfce1e27. Latest consensus state: current_term: 1 leader_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } }
I20260620 21:50:18.922307 2404 tablet_replica.cc:442] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:18.922482 2402 tablet_replica.cc:439] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f: TxnStatusTablet state changed. Reason: New leader 120f6d32a66c4ee4a0aa3c35cfce1e27. Latest consensus state: current_term: 1 leader_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } }
I20260620 21:50:18.922600 2402 tablet_replica.cc:442] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:18.924825 2403 tablet_replica.cc:439] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } }
I20260620 21:50:18.924968 2403 tablet_replica.cc:442] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:50:18.925096 2403 tablet_replica.cc:439] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } }
I20260620 21:50:18.925185 2403 tablet_replica.cc:442] T 091d0379c1dc4b5bab0e49014653e289 P 120f6d32a66c4ee4a0aa3c35cfce1e27: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:50:18.925274 2439 mvcc.cc:205] Tried to move back new op lower bound from 7299040128689979392 to 7299040128464605184. Current Snapshot: MvccSnapshot[applied={T|T < 7299040128689979392}]
I20260620 21:50:18.926090 2404 tablet_replica.cc:439] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } }
I20260620 21:50:18.926198 2404 tablet_replica.cc:442] T 091d0379c1dc4b5bab0e49014653e289 P cb15b8c9da5a41198e80a18045f24345: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:18.927898 2402 tablet_replica.cc:439] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb15b8c9da5a41198e80a18045f24345" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38369 } } peers { permanent_uuid: "120f6d32a66c4ee4a0aa3c35cfce1e27" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33391 } } peers { permanent_uuid: "a2481c04e9c74e0c92cb715c8ac2d15f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 43191 } } }
I20260620 21:50:18.928045 2402 tablet_replica.cc:442] T 091d0379c1dc4b5bab0e49014653e289 P a2481c04e9c74e0c92cb715c8ac2d15f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:18.939951 2448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.936349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.0.193.194:33391)
0620 21:50:18.936558 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.936564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.936702 (+ 138us) server_negotiation.cc:407] Connection header received
0620 21:50:18.936860 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.936863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.936913 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.936993 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:18.937306 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.937839 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.938849 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.939124 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.939260 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.939511 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:18.939591 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:18.939668 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.939758 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.939810 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 21:50:18.951915 2448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.947673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:41961 (local address 127.0.193.194:33391)
0620 21:50:18.947802 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:18.947807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:18.948035 (+ 228us) server_negotiation.cc:407] Connection header received
0620 21:50:18.948148 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:18.948151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:18.948193 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:18.948255 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:18.948530 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.949034 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.950579 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.950785 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.950866 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.951286 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:18.951355 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:18.951416 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:18.951531 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 21:50:18.951602 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":51}
I20260620 21:50:18.951963 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:18.947627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:33391 (local address 127.0.193.194:41961)
0620 21:50:18.947923 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:18.947938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:18.948055 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:18.948279 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:18.948282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:18.948288 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:50:18.948414 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.948419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.949158 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.949161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.950455 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:18.950466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:18.950907 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:18.950911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:18.951014 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:18.951067 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:50:18.951444 (+ 377us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:50:18.951451 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:50:18.951514 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 21:50:18.951624 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":22,"thread_start_us":111,"threads_started":1}
I20260620 21:50:18.998349 1885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:18.998533 1885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:19.000873 1885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd9dba5515ba4088a6075d6b0ed1fcc0: Sending DeleteTablet for 3 replicas of tablet a08c6c5cb68849159db6ad0c21ef4bd0
I20260620 21:50:19.001585 2027 tablet_service.cc:1514] Processing DeleteTablet for tablet a08c6c5cb68849159db6ad0c21ef4bd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:18 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:37228
I20260620 21:50:19.001655 2307 tablet_service.cc:1514] Processing DeleteTablet for tablet a08c6c5cb68849159db6ad0c21ef4bd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:18 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:36372
I20260620 21:50:19.001969 2461 tablet_replica.cc:330] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345: stopping tablet replica
I20260620 21:50:19.001945 2163 tablet_service.cc:1514] Processing DeleteTablet for tablet a08c6c5cb68849159db6ad0c21ef4bd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:18 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:46302
I20260620 21:50:19.002116 2461 raft_consensus.cc:2229] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:19.002247 2461 raft_consensus.cc:2258] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:19.002269 2462 tablet_replica.cc:330] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f: stopping tablet replica
I20260620 21:50:19.002381 2462 raft_consensus.cc:2229] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:19.002487 2462 raft_consensus.cc:2258] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:19.002698 2463 tablet_replica.cc:330] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27: stopping tablet replica
I20260620 21:50:19.002804 2463 raft_consensus.cc:2229] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:50:19.003036 2461 ts_tablet_manager.cc:1905] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:19.003132 2463 raft_consensus.cc:2258] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:19.003860 2462 ts_tablet_manager.cc:1905] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:19.006296 2461 ts_tablet_manager.cc:1918] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:50:19.006402 2461 log.cc:1196] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-0/wal/wals/a08c6c5cb68849159db6ad0c21ef4bd0
I20260620 21:50:19.006695 2461 ts_tablet_manager.cc:1939] T a08c6c5cb68849159db6ad0c21ef4bd0 P cb15b8c9da5a41198e80a18045f24345: Deleting consensus metadata
I20260620 21:50:19.006856 2462 ts_tablet_manager.cc:1918] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:50:19.006948 2462 log.cc:1196] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/a08c6c5cb68849159db6ad0c21ef4bd0
I20260620 21:50:19.007256 2462 ts_tablet_manager.cc:1939] T a08c6c5cb68849159db6ad0c21ef4bd0 P a2481c04e9c74e0c92cb715c8ac2d15f: Deleting consensus metadata
I20260620 21:50:19.007445 2463 ts_tablet_manager.cc:1905] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:19.007519 1875 catalog_manager.cc:4841] TS cb15b8c9da5a41198e80a18045f24345 (127.0.193.193:38369): tablet a08c6c5cb68849159db6ad0c21ef4bd0 (table test-table [id=009cbdda8b104ebb8bbd0f2d2b332d75]) successfully deleted
I20260620 21:50:19.007929 1876 catalog_manager.cc:4841] TS a2481c04e9c74e0c92cb715c8ac2d15f (127.0.193.195:43191): tablet a08c6c5cb68849159db6ad0c21ef4bd0 (table test-table [id=009cbdda8b104ebb8bbd0f2d2b332d75]) successfully deleted
I20260620 21:50:19.010411 2463 ts_tablet_manager.cc:1918] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:50:19.010494 2463 log.cc:1196] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/a08c6c5cb68849159db6ad0c21ef4bd0
I20260620 21:50:19.010746 2463 ts_tablet_manager.cc:1939] T a08c6c5cb68849159db6ad0c21ef4bd0 P 120f6d32a66c4ee4a0aa3c35cfce1e27: Deleting consensus metadata
I20260620 21:50:19.011487 1876 catalog_manager.cc:4841] TS 120f6d32a66c4ee4a0aa3c35cfce1e27 (127.0.193.194:33391): tablet a08c6c5cb68849159db6ad0c21ef4bd0 (table test-table [id=009cbdda8b104ebb8bbd0f2d2b332d75]) successfully deleted
Jun 20 21:50:19 dist-test-slave-kk5c krb5kdc[1845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992218, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.193.193@KRBTEST.COM
I20260620 21:50:19.018761 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:19.003773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.0.193.193:38369)
0620 21:50:19.007869 (+ 4096us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:19.007873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:19.007890 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:50:19.007929 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:19.007931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:19.007986 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:19.008082 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:19.009073 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:19.009846 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:19.011093 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:19.011466 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:19.011582 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:19.014863 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:19.014878 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:19.014896 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:19.014927 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:19.016954 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:19.017438 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:19.017444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:19.017446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:19.017545 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:19.017812 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:19.017815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:19.017817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:19.018132 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:19.018321 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:19.018467 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 21:50:19.018587 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4027,"thread_start_us":60,"threads_started":1}
W20260620 21:50:19.019268 2082 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37252
I20260620 21:50:19.029347 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:19.021020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.0.193.254:34783)
0620 21:50:19.021149 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:19.021153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:19.021166 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:50:19.021241 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:19.021243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:19.021282 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:19.021349 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:19.022136 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:19.022686 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:19.023830 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:19.024047 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:19.024137 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:19.025721 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:19.025725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:19.025727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:19.025761 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:19.027409 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:19.027875 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:19.027878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:19.027880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:19.027963 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:19.028287 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:19.028290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:19.028291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:19.028565 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:19.028675 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:19.028982 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 21:50:19.029110 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 21:50:19.029834 1885 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43000
I20260620 21:50:19.031244 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 1962
I20260620 21:50:19.039016 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 2102
I20260620 21:50:19.046193 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 2242
I20260620 21:50:19.052100 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 1858
2026-06-20T21:50:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3194 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:50:19 dist-test-slave-kk5c krb5kdc[2471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:50:19 dist-test-slave-kk5c krb5kdc[2471](info): set up 2 sockets
Jun 20 21:50:19 dist-test-slave-kk5c krb5kdc[2471](info): commencing operation
krb5kdc: starting...
W20260620 21:50:20.119102 775 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:50:20 dist-test-slave-kk5c krb5kdc[2471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:50:20.254613 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:39551
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:39551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:20.369267 2484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:20.369588 2484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:20.369629 2484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:20.373775 2484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:20.373863 2484 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 21:50:20.373883 2484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:20.373901 2484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:20.373919 2484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:20.373934 2484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:20.379709 2484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:39551
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:39551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:20.381206 2484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:20.382298 2484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:20.391498 2489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:20.391669 2484 server_base.cc:1034] running on GCE node
W20260620 21:50:20.391498 2492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:20.391973 2490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:20.392618 2484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:20.393891 2484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:20.395121 2484 hybrid_clock.cc:648] HybridClock initialized: now 1781992220395101 us; error 37 us; skew 500 ppm
Jun 20 21:50:20 dist-test-slave-kk5c krb5kdc[2471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:20.398140 2484 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:50:20.399413 2484 webserver.cc:460] Webserver started at http://127.0.193.254:42923/ using document root <none> and password file <none>
I20260620 21:50:20.400012 2484 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:20.400087 2484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:20.400266 2484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:20.402376 2484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "7119608480e745a9bd1b9328149af287"
format_stamp: "Formatted at 2026-06-20 21:50:20 on dist-test-slave-kk5c"
server_key: "b0b293a39c0933bb7102b7d7d6034860"
server_key_iv: "329e38aa025fe0c1ecd7d0cc163d4100"
server_key_version: "encryptionkey@0"
I20260620 21:50:20.402909 2484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "7119608480e745a9bd1b9328149af287"
format_stamp: "Formatted at 2026-06-20 21:50:20 on dist-test-slave-kk5c"
server_key: "b0b293a39c0933bb7102b7d7d6034860"
server_key_iv: "329e38aa025fe0c1ecd7d0cc163d4100"
server_key_version: "encryptionkey@0"
I20260620 21:50:20.406906 2484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 21:50:20.409754 2499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:20.411557 2484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 21:50:20.411698 2484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "7119608480e745a9bd1b9328149af287"
format_stamp: "Formatted at 2026-06-20 21:50:20 on dist-test-slave-kk5c"
server_key: "b0b293a39c0933bb7102b7d7d6034860"
server_key_iv: "329e38aa025fe0c1ecd7d0cc163d4100"
server_key_version: "encryptionkey@0"
I20260620 21:50:20.411821 2484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:20.431300 2484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:20.434679 2484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:20.434849 2484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:20.442981 2484 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:39551
I20260620 21:50:20.442996 2561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:39551 every 8 connection(s)
I20260620 21:50:20.444054 2484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:20.447415 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:20.451777 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 2484
I20260620 21:50:20.451920 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:20.452180 775 external_mini_cluster.cc:1428] Setting key 9a98b989b62319915b289dfdfc29624a
I20260620 21:50:20.454332 2562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287: Bootstrap starting.
I20260620 21:50:20.456884 2562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:20.457702 2562 log.cc:824] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:20.459787 2562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287: No bootstrap required, opened a new log
Jun 20 21:50:20 dist-test-slave-kk5c krb5kdc[2471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:20.464144 2562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7119608480e745a9bd1b9328149af287" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39551 } }
I20260620 21:50:20.464555 2562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:20.464638 2562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7119608480e745a9bd1b9328149af287, State: Initialized, Role: FOLLOWER
I20260620 21:50:20.465138 2562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7119608480e745a9bd1b9328149af287" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39551 } }
I20260620 21:50:20.465277 2562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:20.465368 2562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:20.465474 2562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:20.466799 2562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7119608480e745a9bd1b9328149af287" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39551 } }
I20260620 21:50:20.467228 2562 leader_election.cc:302] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7119608480e745a9bd1b9328149af287; no voters:
I20260620 21:50:20.467650 2562 leader_election.cc:288] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:20.467772 2567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:20.468076 2567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [term 1 LEADER]: Becoming Leader. State: Replica: 7119608480e745a9bd1b9328149af287, State: Running, Role: LEADER
I20260620 21:50:20.468485 2567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7119608480e745a9bd1b9328149af287" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39551 } }
I20260620 21:50:20.469096 2562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:20.469280 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:20.453812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.0.193.254:39551)
0620 21:50:20.454603 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:20.454624 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:20.454666 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:50:20.455441 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:20.455463 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:20.455829 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:20.456205 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:20.457062 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:20.458145 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:20.459088 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:20.459708 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:20.460075 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:20.462986 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:20.463000 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:20.463017 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:20.463051 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:20.465322 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:20.466004 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:20.466010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:20.466015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:20.466200 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:20.466526 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:20.466529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:20.466530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:20.467093 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:20.467326 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:20.467720 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 21:50:20.468136 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":192,"threads_started":1}
I20260620 21:50:20.470369 2569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7119608480e745a9bd1b9328149af287. Latest consensus state: current_term: 1 leader_uuid: "7119608480e745a9bd1b9328149af287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7119608480e745a9bd1b9328149af287" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39551 } } }
I20260620 21:50:20.470516 2569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:20.470772 2568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7119608480e745a9bd1b9328149af287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7119608480e745a9bd1b9328149af287" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39551 } } }
I20260620 21:50:20.470868 2568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:20.471655 2576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:20.475204 2576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:20.481307 2576 catalog_manager.cc:1347] Generated new cluster ID: 885f12b11b1c478cb01cf7863d056c47
I20260620 21:50:20.481382 2576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:20.501919 2576 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:20.503052 2576 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:20.517390 2576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287: Generated new TSK 0
I20260620 21:50:20.518278 2576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:50:20.552464 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:39551
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:20.683871 2588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:20.684247 2588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:20.684340 2588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:20.688695 2588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:20.688791 2588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:20.688875 2588 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 21:50:20.688918 2588 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:50:20.688982 2588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:20.695052 2588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.193.254:39551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:20.696456 2588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:20.697741 2588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:20.708528 2593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:20.708801 2594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:20.708892 2596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:20.709656 2588 server_base.cc:1034] running on GCE node
I20260620 21:50:20.710122 2588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:20.710916 2588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:20.712224 2588 hybrid_clock.cc:648] HybridClock initialized: now 1781992220712216 us; error 124 us; skew 500 ppm
Jun 20 21:50:20 dist-test-slave-kk5c krb5kdc[2471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:20.715427 2588 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:50:20.716501 2588 webserver.cc:460] Webserver started at http://127.0.193.193:38415/ using document root <none> and password file <none>
I20260620 21:50:20.717125 2588 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:20.717221 2588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:20.717458 2588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:20.719724 2588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "65dfaf8db4194fbdb570567fa4eb3dfe"
format_stamp: "Formatted at 2026-06-20 21:50:20 on dist-test-slave-kk5c"
server_key: "c8475c035bfcdb9ff7042835e3d7123e"
server_key_iv: "f61031be5a39c1ca4726f0265ef7b4d9"
server_key_version: "encryptionkey@0"
I20260620 21:50:20.720322 2588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "65dfaf8db4194fbdb570567fa4eb3dfe"
format_stamp: "Formatted at 2026-06-20 21:50:20 on dist-test-slave-kk5c"
server_key: "c8475c035bfcdb9ff7042835e3d7123e"
server_key_iv: "f61031be5a39c1ca4726f0265ef7b4d9"
server_key_version: "encryptionkey@0"
I20260620 21:50:20.724402 2588 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:50:20.727367 2603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:20.728734 2588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:20.728880 2588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "65dfaf8db4194fbdb570567fa4eb3dfe"
format_stamp: "Formatted at 2026-06-20 21:50:20 on dist-test-slave-kk5c"
server_key: "c8475c035bfcdb9ff7042835e3d7123e"
server_key_iv: "f61031be5a39c1ca4726f0265ef7b4d9"
server_key_version: "encryptionkey@0"
I20260620 21:50:20.729012 2588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:20.746310 2588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:20.750128 2588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:20.750346 2588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:20.751966 2588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:20.752043 2588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:20.752116 2588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:20.752157 2588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:50:20 dist-test-slave-kk5c krb5kdc[2471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:20.763262 2588 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:37635
I20260620 21:50:20.763334 2718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:37635 every 8 connection(s)
I20260620 21:50:20.764401 2588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:20.766429 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:20.752813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:42999 (local address 127.0.193.254:39551)
0620 21:50:20.752966 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:20.752971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:20.753938 (+ 967us) server_negotiation.cc:407] Connection header received
0620 21:50:20.755072 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:20.755075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:20.755139 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:20.755365 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:20.756715 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:20.757264 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:20.758445 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:20.758765 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:20.758882 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:20.762238 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:20.762242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:20.762244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:20.762272 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:20.764102 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:20.764741 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:20.764748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:20.764751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:20.764859 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:20.765274 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:20.765278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:20.765280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:20.765497 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:20.765612 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:20.766120 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 21:50:20.766256 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:50:20.767284 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:20.753270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39551 (local address 127.0.193.193:42999)
0620 21:50:20.753756 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:20.753807 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:20.754799 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:20.755578 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:20.755586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:20.756012 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:20.756538 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:20.756554 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:20.757398 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:20.757402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:20.758311 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:20.758321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:20.758932 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:20.758936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:20.759158 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:20.759871 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:20.759888 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:20.762005 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:20.764234 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:20.764239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:20.764250 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:20.764621 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:20.764997 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:20.765000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:20.765002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:20.765183 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:20.765609 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:20.765617 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:20.765872 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 21:50:20.766079 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 21:50:20.766404 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":167,"threads_started":1}
I20260620 21:50:20.768606 2719 heartbeater.cc:347] Connected to a master server at 127.0.193.254:39551
I20260620 21:50:20.768770 2719 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:20.769258 2719 heartbeater.cc:511] Master 127.0.193.254:39551 requested a full tablet report, sending...
I20260620 21:50:20.770593 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 2588
I20260620 21:50:20.770704 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:50:20.771013 775 external_mini_cluster.cc:1428] Setting key e26d762971d6f1b5dd2e021fc9fd3814
I20260620 21:50:20.771049 2515 ts_manager.cc:194] Registered new tserver with Master: 65dfaf8db4194fbdb570567fa4eb3dfe (127.0.193.193:37635)
I20260620 21:50:20.772533 2515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:42999
I20260620 21:50:20.779659 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:20.771102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:58559 (local address 127.0.193.254:39551)
0620 21:50:20.771322 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:20.771328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:20.771494 (+ 166us) server_negotiation.cc:407] Connection header received
0620 21:50:20.771703 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:20.771706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:20.771754 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:20.771844 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:20.772348 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:20.773051 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:20.774186 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:20.774574 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:20.774658 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:20.776020 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:20.776023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:20.776026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:20.776049 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:20.777534 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:20.778165 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:20.778168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:20.778170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:20.778336 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:20.778701 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:20.778704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:20.778706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:20.778898 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:20.778984 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:20.779401 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 21:50:20.779523 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":138}
I20260620 21:50:20.779717 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:20.771054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39551 (local address 127.0.193.193:58559)
0620 21:50:20.771397 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:20.771412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:20.771601 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:20.771885 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:20.771888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:20.772114 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:20.772253 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:20.772263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:20.773164 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:20.773167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:20.774044 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:20.774055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:20.774812 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:20.774815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:20.774906 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:20.775520 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:20.775531 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:20.775879 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:20.777704 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:20.777710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:20.777714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:20.778026 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:20.778441 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:20.778446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:20.778448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:20.778568 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:20.779003 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:20.779009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:20.779122 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 21:50:20.779270 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 21:50:20.779448 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":4,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:50:20.801855 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:39551
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:20.926136 2728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:20.926476 2728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:20.926532 2728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:20.931728 2728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:20.931874 2728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:20.931950 2728 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 21:50:20.931975 2728 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:50:20.932014 2728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:20.938637 2728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.193.254:39551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:20.939814 2728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:20.940851 2728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:50:20.951841 2728 server_base.cc:1034] running on GCE node
W20260620 21:50:20.951800 2736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:20.951800 2733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:20.951818 2734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:20.952451 2728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:20.953133 2728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:20.954314 2728 hybrid_clock.cc:648] HybridClock initialized: now 1781992220954277 us; error 53 us; skew 500 ppm
Jun 20 21:50:20 dist-test-slave-kk5c krb5kdc[2471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:20.957515 2728 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:50:20.958675 2728 webserver.cc:460] Webserver started at http://127.0.193.194:33081/ using document root <none> and password file <none>
I20260620 21:50:20.959334 2728 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:20.959406 2728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:20.959631 2728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:20.961819 2728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "d430585cc82548499c57a24dcd70308f"
format_stamp: "Formatted at 2026-06-20 21:50:20 on dist-test-slave-kk5c"
server_key: "2261c0f9a8fdfd5bd377f0831711092e"
server_key_iv: "b033d8f95f436e19da646fa365cff87b"
server_key_version: "encryptionkey@0"
I20260620 21:50:20.962400 2728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "d430585cc82548499c57a24dcd70308f"
format_stamp: "Formatted at 2026-06-20 21:50:20 on dist-test-slave-kk5c"
server_key: "2261c0f9a8fdfd5bd377f0831711092e"
server_key_iv: "b033d8f95f436e19da646fa365cff87b"
server_key_version: "encryptionkey@0"
I20260620 21:50:20.966466 2728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:50:20.969765 2743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:20.971602 2728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:20.971779 2728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "d430585cc82548499c57a24dcd70308f"
format_stamp: "Formatted at 2026-06-20 21:50:20 on dist-test-slave-kk5c"
server_key: "2261c0f9a8fdfd5bd377f0831711092e"
server_key_iv: "b033d8f95f436e19da646fa365cff87b"
server_key_version: "encryptionkey@0"
I20260620 21:50:20.971923 2728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:20.987391 2728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:20.991297 2728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:20.991520 2728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:20.993047 2728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:20.993101 2728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:20.993142 2728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:20.993199 2728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:50:21 dist-test-slave-kk5c krb5kdc[2471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:21.005333 2728 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:36851
I20260620 21:50:21.005470 2858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:36851 every 8 connection(s)
I20260620 21:50:21.006322 2728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:21.009303 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 2728
I20260620 21:50:21.009428 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:50:21.009632 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:20.994192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:43383 (local address 127.0.193.254:39551)
0620 21:50:20.994352 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:20.994357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:20.995250 (+ 893us) server_negotiation.cc:407] Connection header received
0620 21:50:20.996479 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:20.996482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:20.996552 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:20.996663 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:20.998474 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:20.999114 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.000385 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.000608 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.000736 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.005056 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:21.005063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:21.005066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:21.005113 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:21.007010 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:21.007706 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:21.007709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:21.007711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:21.007800 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:21.008249 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:21.008252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:21.008254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:21.008558 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:21.008661 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.009299 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.009455 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:50:21.009778 775 external_mini_cluster.cc:1428] Setting key 084bead382d7d771f95ddaa93d3b2304
I20260620 21:50:21.010569 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:20.994582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39551 (local address 127.0.193.194:43383)
0620 21:50:20.995046 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:20.995108 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:20.996217 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:20.996910 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:20.996923 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:20.997476 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:20.998274 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:20.998299 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:20.999293 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:20.999297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.000253 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.000266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.000794 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.000798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.000989 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.002290 (+ 1301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:21.002322 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:21.004767 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:21.007231 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:21.007237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:21.007252 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:21.007579 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:21.007914 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:21.007917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:21.007918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:21.008077 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:21.008707 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:21.008713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:21.008993 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.009174 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.009631 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":140,"threads_started":1}
I20260620 21:50:21.011960 2859 heartbeater.cc:347] Connected to a master server at 127.0.193.254:39551
I20260620 21:50:21.012133 2859 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:21.012823 2859 heartbeater.cc:511] Master 127.0.193.254:39551 requested a full tablet report, sending...
I20260620 21:50:21.014076 2515 ts_manager.cc:194] Registered new tserver with Master: d430585cc82548499c57a24dcd70308f (127.0.193.194:36851)
I20260620 21:50:21.014971 2515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:43383
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
I20260620 21:50:21.022436 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.014543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:55081 (local address 127.0.193.254:39551)
0620 21:50:21.014665 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.014670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.015065 (+ 395us) server_negotiation.cc:407] Connection header received
0620 21:50:21.015344 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.015349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.015400 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.015453 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:21.016057 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.016651 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.017539 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.017859 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.017969 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.019140 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:21.019143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:21.019146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:21.019172 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:21.020723 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:21.021161 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:21.021165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:21.021167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:21.021250 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:21.021531 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:21.021535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:21.021537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:21.021783 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:21.021897 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.022161 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.022271 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:50:21.022604 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.014657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39551 (local address 127.0.193.194:55081)
0620 21:50:21.014971 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.014983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.015165 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.015594 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.015597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.015826 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:21.015931 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.015937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.016784 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.016790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.017418 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.017427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.017976 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.017980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.018080 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.018633 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:21.018651 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:21.018988 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:21.020835 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:21.020841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:21.020843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:21.021056 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:21.021345 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:21.021348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:21.021350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:21.021434 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:21.021885 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:21.021888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:21.022000 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.022108 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.022269 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:50:21.041983 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:39551
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:21.160619 2868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:21.160936 2868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:21.161017 2868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:21.165391 2868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:21.165499 2868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:21.165570 2868 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 21:50:21.165611 2868 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:50:21.165676 2868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:21.171591 2868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.193.254:39551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:21.172724 2868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:21.173785 2868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:21.183894 2876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:21.183955 2874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:21.184054 2873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:21.184319 2868 server_base.cc:1034] running on GCE node
I20260620 21:50:21.184783 2868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:21.185417 2868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:21.186632 2868 hybrid_clock.cc:648] HybridClock initialized: now 1781992221186600 us; error 54 us; skew 500 ppm
Jun 20 21:50:21 dist-test-slave-kk5c krb5kdc[2471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:21.189850 2868 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:50:21.190862 2868 webserver.cc:460] Webserver started at http://127.0.193.195:40451/ using document root <none> and password file <none>
I20260620 21:50:21.191483 2868 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:21.191569 2868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:21.191797 2868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:21.193986 2868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "6e5ccbdee09e481683ebfee375ad6f59"
format_stamp: "Formatted at 2026-06-20 21:50:21 on dist-test-slave-kk5c"
server_key: "47fdac93e28ed4c655da7df04bcb2b40"
server_key_iv: "0981f2767e591152cb92915e6210c243"
server_key_version: "encryptionkey@0"
I20260620 21:50:21.194584 2868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "6e5ccbdee09e481683ebfee375ad6f59"
format_stamp: "Formatted at 2026-06-20 21:50:21 on dist-test-slave-kk5c"
server_key: "47fdac93e28ed4c655da7df04bcb2b40"
server_key_iv: "0981f2767e591152cb92915e6210c243"
server_key_version: "encryptionkey@0"
I20260620 21:50:21.198665 2868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:50:21.201447 2883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:21.202744 2868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 21:50:21.202878 2868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "6e5ccbdee09e481683ebfee375ad6f59"
format_stamp: "Formatted at 2026-06-20 21:50:21 on dist-test-slave-kk5c"
server_key: "47fdac93e28ed4c655da7df04bcb2b40"
server_key_iv: "0981f2767e591152cb92915e6210c243"
server_key_version: "encryptionkey@0"
I20260620 21:50:21.203001 2868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:21.213973 2868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:21.217433 2868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:21.217640 2868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:21.219056 2868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:21.219128 2868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:21.219218 2868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:21.219280 2868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:50:21 dist-test-slave-kk5c krb5kdc[2471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:21.229915 2868 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:40265
I20260620 21:50:21.229979 2998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:40265 every 8 connection(s)
I20260620 21:50:21.230810 2868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:50:21.234019 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.220016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:43153 (local address 127.0.193.254:39551)
0620 21:50:21.220169 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.220174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.221098 (+ 924us) server_negotiation.cc:407] Connection header received
0620 21:50:21.222232 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.222235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.222294 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.222398 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:21.224160 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.224855 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.226802 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.227070 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.227162 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.230373 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:21.230377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:21.230379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:21.230404 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:21.231949 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:21.232525 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:21.232529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:21.232531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:21.232612 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:21.232947 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:21.232950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:21.232952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:21.233141 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:21.233245 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.233716 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.233837 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:50:21.234791 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.220472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39551 (local address 127.0.193.195:43153)
0620 21:50:21.220920 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.220969 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.221962 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.222667 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.222679 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.223174 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:21.223968 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.223991 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.224974 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.224979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.226631 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.226654 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.227277 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.227283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.227531 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.228240 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:21.228269 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:21.230197 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:21.232068 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:21.232074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:21.232088 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:21.232420 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:21.232707 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:21.232712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:21.232714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:21.232850 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:21.233242 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:21.233248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:21.233492 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.233679 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.233988 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":128,"threads_started":1}
I20260620 21:50:21.235956 2999 heartbeater.cc:347] Connected to a master server at 127.0.193.254:39551
I20260620 21:50:21.236109 2999 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:21.236671 2999 heartbeater.cc:511] Master 127.0.193.254:39551 requested a full tablet report, sending...
I20260620 21:50:21.237818 2516 ts_manager.cc:194] Registered new tserver with Master: 6e5ccbdee09e481683ebfee375ad6f59 (127.0.193.195:40265)
I20260620 21:50:21.238497 2516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:43153
I20260620 21:50:21.238999 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 2868
I20260620 21:50:21.239130 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:50:21.239476 775 external_mini_cluster.cc:1428] Setting key 6dd786b9c8a4feec7ff057da61e1016a
I20260620 21:50:21.241873 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:50:21.246616 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.238513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:50365 (local address 127.0.193.254:39551)
0620 21:50:21.238641 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.238646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.238910 (+ 264us) server_negotiation.cc:407] Connection header received
0620 21:50:21.239093 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.239097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.239152 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.239261 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:21.239854 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.240543 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.241370 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.241705 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.241815 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.243101 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:21.243107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:21.243114 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:21.243145 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:21.244644 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:21.245182 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:21.245185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:21.245186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:21.245257 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:21.245573 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:21.245575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:21.245577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:21.245923 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:21.246023 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.246392 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.246493 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:50:21.246776 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.238378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39551 (local address 127.0.193.195:50365)
0620 21:50:21.238830 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.238846 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.238972 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.239359 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.239362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.239597 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:21.239708 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.239715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.240666 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.240670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.241269 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.241276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.241820 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.241823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.241909 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.242593 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:21.242605 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:21.242957 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:21.244758 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:21.244761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:21.244765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:21.245082 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:21.245345 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:21.245352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:21.245357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:21.245470 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:21.246048 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:21.246055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:21.246189 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.246326 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.246522 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":123,"threads_started":1}
Jun 20 21:50:21 dist-test-slave-kk5c krb5kdc[2471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:50:21 dist-test-slave-kk5c krb5kdc[2471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:21.270843 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.260504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.0.193.254:39551)
0620 21:50:21.260754 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.260758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.260773 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:50:21.260815 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.260818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.260858 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.260938 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:21.261806 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.262346 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.263700 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.263936 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.264039 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.266970 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:21.266974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:21.266976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:21.267007 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:21.268584 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:21.269189 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:21.269193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:21.269194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:21.269285 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:21.269778 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:21.269781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:21.269782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:21.270120 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:21.270240 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.270564 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.270678 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260620 21:50:21.274206 2516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:50:21.277913 2516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:21.292946 3016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.286660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:37635 (local address 127.0.0.1:46714)
0620 21:50:21.287379 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.287432 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.287560 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.288309 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.288314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.288332 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:21.288477 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.288486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.289845 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.289850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.291488 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.291499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.292326 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.292331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.292472 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.292537 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.292655 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.292796 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":136,"thread_start_us":94,"threads_started":1}
I20260620 21:50:21.292941 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.286768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.0.193.193:37635)
0620 21:50:21.287462 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.287469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.287497 (+ 28us) server_negotiation.cc:407] Connection header received
0620 21:50:21.287753 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.287759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.288025 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.288151 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:21.288601 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.289726 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.291602 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.292202 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.292288 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.292398 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.292696 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.292795 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":98,"threads_started":1}
I20260620 21:50:21.293999 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.287350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:40265 (local address 127.0.0.1:54402)
0620 21:50:21.288324 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.288336 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.288434 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.288837 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.288840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.288855 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:21.288939 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.288947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.290001 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.290004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.292451 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.292459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.293527 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.293531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.293645 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.293687 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.293767 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.293852 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":963,"thread_start_us":94,"threads_started":1}
I20260620 21:50:21.294415 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.286878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.0.193.194:36851)
0620 21:50:21.287991 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.288003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.288026 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:50:21.288101 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.288107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.288307 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.288458 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:21.288820 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.289884 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.291753 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.292438 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.292565 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.292789 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.294065 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.294281 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1016,"thread_start_us":99,"threads_started":1}
I20260620 21:50:21.294476 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.287758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.0.193.195:40265)
0620 21:50:21.288195 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.288205 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.288444 (+ 239us) server_negotiation.cc:407] Connection header received
0620 21:50:21.288530 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.288537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.288715 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.288852 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:21.289040 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.289733 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.292572 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.293383 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.293512 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.293662 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.294200 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.294325 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":95,"threads_started":1}
I20260620 21:50:21.295408 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.286660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:36851 (local address 127.0.0.1:33264)
0620 21:50:21.287624 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.287639 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.287731 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.288608 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.288610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.288626 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:21.288710 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.288716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.289972 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.289975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.291603 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.291613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.293724 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.293728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.293823 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.293847 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.293919 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.293990 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":219,"thread_start_us":88,"threads_started":1}
I20260620 21:50:21.296300 2653 tablet_service.cc:1467] Processing CreateTablet for tablet 8b70a8b34166497e8e8a9e3efc367de3 (DEFAULT_TABLE table=test-table [id=56523b4f7e77455aaa898de70f1e0764]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:21.297297 2653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b70a8b34166497e8e8a9e3efc367de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:21.297277 2933 tablet_service.cc:1467] Processing CreateTablet for tablet 8b70a8b34166497e8e8a9e3efc367de3 (DEFAULT_TABLE table=test-table [id=56523b4f7e77455aaa898de70f1e0764]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:21.298379 2933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b70a8b34166497e8e8a9e3efc367de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:21.298116 2793 tablet_service.cc:1467] Processing CreateTablet for tablet 8b70a8b34166497e8e8a9e3efc367de3 (DEFAULT_TABLE table=test-table [id=56523b4f7e77455aaa898de70f1e0764]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:21.299085 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b70a8b34166497e8e8a9e3efc367de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:21.304450 3022 tablet_bootstrap.cc:492] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe: Bootstrap starting.
I20260620 21:50:21.306882 3023 tablet_bootstrap.cc:492] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f: Bootstrap starting.
I20260620 21:50:21.307067 3022 tablet_bootstrap.cc:654] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:21.307847 3024 tablet_bootstrap.cc:492] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59: Bootstrap starting.
I20260620 21:50:21.308101 3022 log.cc:824] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:21.310190 3023 tablet_bootstrap.cc:654] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:21.310185 3024 tablet_bootstrap.cc:654] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:21.311108 3023 log.cc:824] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:21.311106 3024 log.cc:824] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:21.313133 3022 tablet_bootstrap.cc:492] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe: No bootstrap required, opened a new log
I20260620 21:50:21.313130 3023 tablet_bootstrap.cc:492] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f: No bootstrap required, opened a new log
I20260620 21:50:21.313366 3023 ts_tablet_manager.cc:1397] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260620 21:50:21.313367 3022 ts_tablet_manager.cc:1397] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 21:50:21.313635 3024 tablet_bootstrap.cc:492] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59: No bootstrap required, opened a new log
I20260620 21:50:21.313853 3024 ts_tablet_manager.cc:1397] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:50:21.317521 3024 raft_consensus.cc:348] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.317799 3024 raft_consensus.cc:374] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:21.317602 3023 raft_consensus.cc:348] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.317854 3024 raft_consensus.cc:729] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5ccbdee09e481683ebfee375ad6f59, State: Initialized, Role: FOLLOWER
I20260620 21:50:21.317869 3023 raft_consensus.cc:374] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:21.317924 3023 raft_consensus.cc:729] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d430585cc82548499c57a24dcd70308f, State: Initialized, Role: FOLLOWER
I20260620 21:50:21.317996 3022 raft_consensus.cc:348] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.318249 3022 raft_consensus.cc:374] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:21.318305 3022 raft_consensus.cc:729] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65dfaf8db4194fbdb570567fa4eb3dfe, State: Initialized, Role: FOLLOWER
I20260620 21:50:21.318353 3024 consensus_queue.cc:260] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.318404 3023 consensus_queue.cc:260] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.318677 3022 consensus_queue.cc:260] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.319424 2859 heartbeater.cc:503] Master 127.0.193.254:39551 was elected leader, sending a full tablet report...
I20260620 21:50:21.319543 3024 ts_tablet_manager.cc:1428] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:50:21.319537 3022 ts_tablet_manager.cc:1428] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260620 21:50:21.319537 3023 ts_tablet_manager.cc:1428] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 21:50:21.319890 2719 heartbeater.cc:503] Master 127.0.193.254:39551 was elected leader, sending a full tablet report...
I20260620 21:50:21.320142 2999 heartbeater.cc:503] Master 127.0.193.254:39551 was elected leader, sending a full tablet report...
I20260620 21:50:21.379382 3030 raft_consensus.cc:482] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:21.379652 3030 raft_consensus.cc:504] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.381003 3030 leader_election.cc:288] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d430585cc82548499c57a24dcd70308f (127.0.193.194:36851), 6e5ccbdee09e481683ebfee375ad6f59 (127.0.193.195:40265)
I20260620 21:50:21.385421 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.381503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:60079 (local address 127.0.193.195:40265)
0620 21:50:21.381636 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.381640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.381858 (+ 218us) server_negotiation.cc:407] Connection header received
0620 21:50:21.382008 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.382010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.382216 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.382322 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:21.382601 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.383099 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.384401 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.384786 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.384883 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.384975 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.385213 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.385284 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:50:21.385432 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.381361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:60069 (local address 127.0.193.194:36851)
0620 21:50:21.381521 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.381526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.381773 (+ 247us) server_negotiation.cc:407] Connection header received
0620 21:50:21.381967 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.381970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.382024 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.382086 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:21.382464 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.383056 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.384352 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.384823 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.384979 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.385035 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.385211 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.385289 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:50:21.385452 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.381232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:36851 (local address 127.0.193.193:60069)
0620 21:50:21.381651 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.381670 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.381825 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.382164 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.382167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.382185 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:21.382343 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.382350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.383280 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.383287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.384209 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.384219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.384976 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.384980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.385086 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.385103 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.385193 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.385279 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":49,"thread_start_us":125,"threads_started":1}
I20260620 21:50:21.385460 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.381324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:40265 (local address 127.0.193.193:60079)
0620 21:50:21.381771 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.381783 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.381899 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.382328 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.382330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.382344 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:21.382478 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.382486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.383345 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.383348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.384242 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.384251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.384912 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.384917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.385073 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.385101 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.385196 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.385289 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":4,"thread_start_us":126,"threads_started":1}
I20260620 21:50:21.386220 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b70a8b34166497e8e8a9e3efc367de3" candidate_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d430585cc82548499c57a24dcd70308f" is_pre_election: true
I20260620 21:50:21.386389 2953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b70a8b34166497e8e8a9e3efc367de3" candidate_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5ccbdee09e481683ebfee375ad6f59" is_pre_election: true
I20260620 21:50:21.386570 2813 raft_consensus.cc:2454] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65dfaf8db4194fbdb570567fa4eb3dfe in term 0.
I20260620 21:50:21.386703 2953 raft_consensus.cc:2454] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65dfaf8db4194fbdb570567fa4eb3dfe in term 0.
I20260620 21:50:21.387102 2604 leader_election.cc:302] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65dfaf8db4194fbdb570567fa4eb3dfe, d430585cc82548499c57a24dcd70308f; no voters:
I20260620 21:50:21.387419 3030 raft_consensus.cc:2781] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:21.387506 3030 raft_consensus.cc:482] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:21.387557 3030 raft_consensus.cc:3037] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:21.388640 3030 raft_consensus.cc:504] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.389020 3030 leader_election.cc:288] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [CANDIDATE]: Term 1 election: Requested vote from peers d430585cc82548499c57a24dcd70308f (127.0.193.194:36851), 6e5ccbdee09e481683ebfee375ad6f59 (127.0.193.195:40265)
I20260620 21:50:21.389370 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b70a8b34166497e8e8a9e3efc367de3" candidate_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d430585cc82548499c57a24dcd70308f"
I20260620 21:50:21.389389 2953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b70a8b34166497e8e8a9e3efc367de3" candidate_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5ccbdee09e481683ebfee375ad6f59"
I20260620 21:50:21.389518 2953 raft_consensus.cc:3037] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:21.389564 2813 raft_consensus.cc:3037] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:21.390697 2813 raft_consensus.cc:2454] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65dfaf8db4194fbdb570567fa4eb3dfe in term 1.
I20260620 21:50:21.390746 2953 raft_consensus.cc:2454] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65dfaf8db4194fbdb570567fa4eb3dfe in term 1.
I20260620 21:50:21.391058 2605 leader_election.cc:302] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65dfaf8db4194fbdb570567fa4eb3dfe, 6e5ccbdee09e481683ebfee375ad6f59; no voters:
I20260620 21:50:21.391263 3030 raft_consensus.cc:2781] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:21.391507 3030 raft_consensus.cc:686] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 LEADER]: Becoming Leader. State: Replica: 65dfaf8db4194fbdb570567fa4eb3dfe, State: Running, Role: LEADER
I20260620 21:50:21.391850 3030 consensus_queue.cc:237] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.395427 2514 catalog_manager.cc:5495] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe reported cstate change: term changed from 0 to 1, leader changed from <none> to 65dfaf8db4194fbdb570567fa4eb3dfe (127.0.193.193). New cstate: current_term: 1 leader_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } health_report { overall_health: UNKNOWN } } }
Jun 20 21:50:21 dist-test-slave-kk5c krb5kdc[2471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:21.446929 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.437249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53862 (local address 127.0.193.254:39551)
0620 21:50:21.437400 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.437409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.437493 (+ 84us) server_negotiation.cc:407] Connection header received
0620 21:50:21.437726 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.437732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.437786 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.437885 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:21.438574 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.439175 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.440067 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.440353 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.440468 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.443157 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:21.443161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:21.443163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:21.443238 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:21.445057 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:21.445584 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:21.445589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:21.445591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:21.445673 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:21.445945 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:21.445950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:21.445952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:21.446179 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:21.446282 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.446572 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.446711 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:50:21.446934 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.437078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39551 (local address 127.0.0.1:53862)
0620 21:50:21.437408 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.437423 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.437531 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.437916 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.437919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.438358 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:21.438456 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.438463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.439336 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.439339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.439911 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.439920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.440471 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.440474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.440547 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.441372 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:21.441385 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:21.443011 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:21.445223 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:21.445232 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:21.445239 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:21.445480 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:21.445771 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:21.445776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:21.445779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:21.445859 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:21.446289 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:21.446298 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:21.446408 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.446525 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.446784 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":89,"thread_start_us":113,"threads_started":1}
I20260620 21:50:21.451156 2514 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:53862:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 21:50:21.451962 2514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:21.458446 2933 tablet_service.cc:1467] Processing CreateTablet for tablet ec33c845f04c4d319a6f037dae6f28d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09c412b09a1044388b73d879626c62f4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:50:21.458357 2653 tablet_service.cc:1467] Processing CreateTablet for tablet ec33c845f04c4d319a6f037dae6f28d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09c412b09a1044388b73d879626c62f4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:50:21.458523 2793 tablet_service.cc:1467] Processing CreateTablet for tablet ec33c845f04c4d319a6f037dae6f28d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09c412b09a1044388b73d879626c62f4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:50:21.459159 2653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec33c845f04c4d319a6f037dae6f28d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:21.459159 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec33c845f04c4d319a6f037dae6f28d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:21.459256 2933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec33c845f04c4d319a6f037dae6f28d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:21.461807 3024 tablet_bootstrap.cc:492] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59: Bootstrap starting.
I20260620 21:50:21.462402 3023 tablet_bootstrap.cc:492] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f: Bootstrap starting.
I20260620 21:50:21.462805 3024 tablet_bootstrap.cc:654] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:21.463289 3023 tablet_bootstrap.cc:654] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:21.464200 3024 tablet_bootstrap.cc:492] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59: No bootstrap required, opened a new log
I20260620 21:50:21.464301 3024 ts_tablet_manager.cc:1397] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 21:50:21.464686 3022 tablet_bootstrap.cc:492] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe: Bootstrap starting.
I20260620 21:50:21.464704 3024 raft_consensus.cc:348] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.464824 3024 raft_consensus.cc:374] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:21.464859 3024 raft_consensus.cc:729] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5ccbdee09e481683ebfee375ad6f59, State: Initialized, Role: FOLLOWER
I20260620 21:50:21.464991 3024 consensus_queue.cc:260] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.465128 3023 tablet_bootstrap.cc:492] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f: No bootstrap required, opened a new log
I20260620 21:50:21.465216 3023 ts_tablet_manager.cc:1397] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 21:50:21.465270 3024 ts_tablet_manager.cc:1428] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:50:21.465241 3029 tablet_replica.cc:439] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } } }
I20260620 21:50:21.465368 3029 tablet_replica.cc:442] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:21.465723 3023 raft_consensus.cc:348] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.465834 3023 raft_consensus.cc:374] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:21.465865 3023 raft_consensus.cc:729] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d430585cc82548499c57a24dcd70308f, State: Initialized, Role: FOLLOWER
I20260620 21:50:21.465983 3023 consensus_queue.cc:260] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.466133 3022 tablet_bootstrap.cc:654] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:21.466293 3028 tablet_replica.cc:439] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } } }
I20260620 21:50:21.466434 3028 tablet_replica.cc:442] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:21.466920 3023 ts_tablet_manager.cc:1428] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 21:50:21.467823 3022 tablet_bootstrap.cc:492] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe: No bootstrap required, opened a new log
I20260620 21:50:21.467942 3022 ts_tablet_manager.cc:1397] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:50:21.468423 3022 raft_consensus.cc:348] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.468531 3022 raft_consensus.cc:374] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:21.468554 3022 raft_consensus.cc:729] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65dfaf8db4194fbdb570567fa4eb3dfe, State: Initialized, Role: FOLLOWER
I20260620 21:50:21.468680 3022 consensus_queue.cc:260] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.468942 3030 tablet_replica.cc:439] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } } }
I20260620 21:50:21.469079 3030 tablet_replica.cc:442] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:21.469357 3022 ts_tablet_manager.cc:1428] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260620 21:50:21.482735 3000 tablet.cc:2369] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:50:21.507601 2860 tablet.cc:2369] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:50:21.515813 2720 tablet.cc:2369] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:50:21.519024 3029 raft_consensus.cc:482] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:21.519227 3029 raft_consensus.cc:504] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.520462 3029 leader_election.cc:288] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65dfaf8db4194fbdb570567fa4eb3dfe (127.0.193.193:37635), d430585cc82548499c57a24dcd70308f (127.0.193.194:36851)
I20260620 21:50:21.524605 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.520772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:53103 (local address 127.0.193.193:37635)
0620 21:50:21.520947 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.520953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.520977 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:50:21.521145 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.521151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.521197 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.521290 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:21.521585 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.522158 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.523587 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.524014 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.524159 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.524218 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.524396 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.524469 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 21:50:21.524605 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.520714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:37635 (local address 127.0.193.195:53103)
0620 21:50:21.520911 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.520927 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.521028 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.521321 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.521325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.521344 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:21.521471 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.521477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.522275 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.522280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.523322 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.523333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.524146 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.524151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.524263 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.524285 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.524380 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.524460 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":38}
I20260620 21:50:21.524747 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.520855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:46363 (local address 127.0.193.194:36851)
0620 21:50:21.520977 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.520982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.521269 (+ 287us) server_negotiation.cc:407] Connection header received
0620 21:50:21.521395 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.521398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.521437 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.521527 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:21.521796 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.522286 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.523530 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.524019 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.524175 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.524236 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.524485 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.524561 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 21:50:21.524760 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.520759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:36851 (local address 127.0.193.195:46363)
0620 21:50:21.521178 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.521188 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.521290 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.521536 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.521539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.521554 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:21.521688 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.521693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.522403 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.522409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.523410 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.523420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.524204 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.524208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.524346 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.524363 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.524467 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.524539 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":116,"threads_started":1}
I20260620 21:50:21.525323 2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec33c845f04c4d319a6f037dae6f28d8" candidate_uuid: "6e5ccbdee09e481683ebfee375ad6f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" is_pre_election: true
I20260620 21:50:21.525436 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec33c845f04c4d319a6f037dae6f28d8" candidate_uuid: "6e5ccbdee09e481683ebfee375ad6f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d430585cc82548499c57a24dcd70308f" is_pre_election: true
I20260620 21:50:21.525596 2813 raft_consensus.cc:2454] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e5ccbdee09e481683ebfee375ad6f59 in term 0.
I20260620 21:50:21.525610 2673 raft_consensus.cc:2454] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e5ccbdee09e481683ebfee375ad6f59 in term 0.
I20260620 21:50:21.525977 2884 leader_election.cc:302] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e5ccbdee09e481683ebfee375ad6f59, d430585cc82548499c57a24dcd70308f; no voters:
I20260620 21:50:21.526340 3029 raft_consensus.cc:2781] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:21.526422 3029 raft_consensus.cc:482] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:21.526468 3029 raft_consensus.cc:3037] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:21.527791 3029 raft_consensus.cc:504] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.528219 3029 leader_election.cc:288] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [CANDIDATE]: Term 1 election: Requested vote from peers 65dfaf8db4194fbdb570567fa4eb3dfe (127.0.193.193:37635), d430585cc82548499c57a24dcd70308f (127.0.193.194:36851)
I20260620 21:50:21.528586 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec33c845f04c4d319a6f037dae6f28d8" candidate_uuid: "6e5ccbdee09e481683ebfee375ad6f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d430585cc82548499c57a24dcd70308f"
I20260620 21:50:21.528618 2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec33c845f04c4d319a6f037dae6f28d8" candidate_uuid: "6e5ccbdee09e481683ebfee375ad6f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe"
I20260620 21:50:21.528759 2813 raft_consensus.cc:3037] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:21.528771 2673 raft_consensus.cc:3037] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:21.529688 2813 raft_consensus.cc:2454] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e5ccbdee09e481683ebfee375ad6f59 in term 1.
I20260620 21:50:21.529845 2673 raft_consensus.cc:2454] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e5ccbdee09e481683ebfee375ad6f59 in term 1.
I20260620 21:50:21.530028 2884 leader_election.cc:302] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e5ccbdee09e481683ebfee375ad6f59, d430585cc82548499c57a24dcd70308f; no voters:
I20260620 21:50:21.530253 3029 raft_consensus.cc:2781] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:21.530506 3029 raft_consensus.cc:686] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 LEADER]: Becoming Leader. State: Replica: 6e5ccbdee09e481683ebfee375ad6f59, State: Running, Role: LEADER
I20260620 21:50:21.530861 3029 consensus_queue.cc:237] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.533239 3045 tablet_replica.cc:439] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59: TxnStatusTablet state changed. Reason: New leader 6e5ccbdee09e481683ebfee375ad6f59. Latest consensus state: current_term: 1 leader_uuid: "6e5ccbdee09e481683ebfee375ad6f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } } }
I20260620 21:50:21.533375 3045 tablet_replica.cc:442] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:50:21.533684 3047 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 21:50:21.533949 3047 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 21:50:21.534715 2515 catalog_manager.cc:5495] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e5ccbdee09e481683ebfee375ad6f59 (127.0.193.195). New cstate: current_term: 1 leader_uuid: "6e5ccbdee09e481683ebfee375ad6f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } health_report { overall_health: HEALTHY } } }
I20260620 21:50:21.556349 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.551456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.0.193.195:40265)
0620 21:50:21.551596 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.551600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.551698 (+ 98us) server_negotiation.cc:407] Connection header received
0620 21:50:21.551889 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.551892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.551961 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.552070 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:21.552356 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.553143 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.554368 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.554758 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.554867 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.555257 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:21.555844 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:21.555961 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.556110 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.556174 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:50:21.556553 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.551324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:40265 (local address 127.0.0.1:54406)
0620 21:50:21.551551 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.551628 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.551746 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.552113 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.552116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.552124 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:50:21.552246 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.552252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.553284 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.553287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.554229 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.554242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.554893 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.554896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.554979 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.555034 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:50:21.555954 (+ 920us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:50:21.555977 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.556086 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.556333 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 21:50:21.562326 2673 raft_consensus.cc:1261] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Refusing update from remote peer 6e5ccbdee09e481683ebfee375ad6f59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:21.562363 2813 raft_consensus.cc:1261] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Refusing update from remote peer 6e5ccbdee09e481683ebfee375ad6f59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:21.563021 3045 consensus_queue.cc:1035] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:21.563259 3029 consensus_queue.cc:1035] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:21.566741 3030 tablet_replica.cc:439] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe: TxnStatusTablet state changed. Reason: New leader 6e5ccbdee09e481683ebfee375ad6f59. Latest consensus state: current_term: 1 leader_uuid: "6e5ccbdee09e481683ebfee375ad6f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } } }
I20260620 21:50:21.566896 3030 tablet_replica.cc:442] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:21.567034 3028 tablet_replica.cc:439] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f: TxnStatusTablet state changed. Reason: New leader 6e5ccbdee09e481683ebfee375ad6f59. Latest consensus state: current_term: 1 leader_uuid: "6e5ccbdee09e481683ebfee375ad6f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } } }
I20260620 21:50:21.567169 3028 tablet_replica.cc:442] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:21.570493 3045 tablet_replica.cc:439] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e5ccbdee09e481683ebfee375ad6f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } } }
I20260620 21:50:21.570636 3045 tablet_replica.cc:442] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:50:21.571002 3029 tablet_replica.cc:439] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e5ccbdee09e481683ebfee375ad6f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } } }
I20260620 21:50:21.571127 3029 tablet_replica.cc:442] T ec33c845f04c4d319a6f037dae6f28d8 P 6e5ccbdee09e481683ebfee375ad6f59: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:50:21.571323 3050 mvcc.cc:205] Tried to move back new op lower bound from 7299040139512127488 to 7299040139393241088. Current Snapshot: MvccSnapshot[applied={T|T < 7299040139512127488}]
I20260620 21:50:21.572610 3028 tablet_replica.cc:439] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e5ccbdee09e481683ebfee375ad6f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } } }
I20260620 21:50:21.572734 3028 tablet_replica.cc:442] T ec33c845f04c4d319a6f037dae6f28d8 P d430585cc82548499c57a24dcd70308f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:21.574815 3030 tablet_replica.cc:439] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e5ccbdee09e481683ebfee375ad6f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } } }
I20260620 21:50:21.574929 3030 tablet_replica.cc:442] T ec33c845f04c4d319a6f037dae6f28d8 P 65dfaf8db4194fbdb570567fa4eb3dfe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:21.584970 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.580693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46730 (local address 127.0.193.193:37635)
0620 21:50:21.580814 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.580819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.580833 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:50:21.580982 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.580985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.581045 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.581164 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:21.581385 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.582180 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.583149 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.583570 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.583686 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.584043 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:21.584499 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:21.584694 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.584759 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.584818 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:50:21.604846 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.600790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:50479 (local address 127.0.193.195:40265)
0620 21:50:21.600946 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.600951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.601193 (+ 242us) server_negotiation.cc:407] Connection header received
0620 21:50:21.601335 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.601338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.601388 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.601460 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:21.601786 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.602352 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.603625 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.603854 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.603944 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.604335 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:21.604418 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:21.604497 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.604690 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.604740 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:50:21.604868 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.600718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:40265 (local address 127.0.193.193:50479)
0620 21:50:21.601095 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.601111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.601237 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.601500 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.601503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.601512 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:50:21.601656 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.601665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.602458 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.602467 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.603498 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.603510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.603970 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.603973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.604072 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.604132 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:50:21.604524 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:50:21.604530 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.604597 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.604683 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":147,"threads_started":1}
I20260620 21:50:21.615370 2813 raft_consensus.cc:1261] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Refusing update from remote peer 65dfaf8db4194fbdb570567fa4eb3dfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:21.616061 3030 consensus_queue.cc:1035] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:21.616916 2953 raft_consensus.cc:1261] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Refusing update from remote peer 65dfaf8db4194fbdb570567fa4eb3dfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:21.617606 3030 consensus_queue.cc:1035] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:21.647136 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.643326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:37635 (local address 127.0.193.195:42105)
0620 21:50:21.643482 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.643500 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.643607 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.643826 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.643829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.643836 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:50:21.643943 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.643949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.644709 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.644712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.645699 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.645710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.646410 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.646414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.646512 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.646573 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:50:21.646897 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:50:21.646901 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.646947 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.647017 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 21:50:21.647212 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.643381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:42105 (local address 127.0.193.193:37635)
0620 21:50:21.643504 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.643509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.643550 (+ 41us) server_negotiation.cc:407] Connection header received
0620 21:50:21.643681 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.643684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.643727 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.643794 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:21.644043 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.644602 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.645846 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.646216 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.646319 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.646724 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:21.646814 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:21.646894 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.646983 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.647048 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 21:50:21.675395 2515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:21.675616 2515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:21.678478 2515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287: Sending DeleteTablet for 3 replicas of tablet 8b70a8b34166497e8e8a9e3efc367de3
I20260620 21:50:21.679523 2793 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b70a8b34166497e8e8a9e3efc367de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:21 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:33264
I20260620 21:50:21.680096 3069 tablet_replica.cc:330] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f: stopping tablet replica
I20260620 21:50:21.680066 2653 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b70a8b34166497e8e8a9e3efc367de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:21 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:46714
I20260620 21:50:21.680583 3069 raft_consensus.cc:2229] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:21.680860 3069 raft_consensus.cc:2258] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:21.681499 2515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:50:21.681958 2515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:21.682790 3069 ts_tablet_manager.cc:1905] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:21.683735 2933 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b70a8b34166497e8e8a9e3efc367de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:21 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:54402
I20260620 21:50:21.686136 3070 tablet_replica.cc:330] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe: stopping tablet replica
I20260620 21:50:21.686532 3070 raft_consensus.cc:2229] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:50:21.686939 3070 raft_consensus.cc:2258] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:21.687717 3071 tablet_replica.cc:330] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59: stopping tablet replica
I20260620 21:50:21.688282 3071 raft_consensus.cc:2229] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:21.688249 2793 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0082fffc76483fbc2b4187d1a37e27 (DEFAULT_TABLE table=test-table [id=de413b095fae4118ab643f39bf036c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:21.688472 3071 raft_consensus.cc:2258] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:21.688562 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0082fffc76483fbc2b4187d1a37e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:21.691375 2933 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0082fffc76483fbc2b4187d1a37e27 (DEFAULT_TABLE table=test-table [id=de413b095fae4118ab643f39bf036c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:21.691715 2933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0082fffc76483fbc2b4187d1a37e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:21.695122 2653 tablet_service.cc:1467] Processing CreateTablet for tablet 6c0082fffc76483fbc2b4187d1a37e27 (DEFAULT_TABLE table=test-table [id=de413b095fae4118ab643f39bf036c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:21.695623 2653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c0082fffc76483fbc2b4187d1a37e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:21.696262 3070 ts_tablet_manager.cc:1905] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:21.700769 3024 tablet_bootstrap.cc:492] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59: Bootstrap starting.
I20260620 21:50:21.701911 3024 tablet_bootstrap.cc:654] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:21.699172 3069 ts_tablet_manager.cc:1918] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:50:21.702170 3069 log.cc:1196] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/8b70a8b34166497e8e8a9e3efc367de3
I20260620 21:50:21.702503 3069 ts_tablet_manager.cc:1939] T 8b70a8b34166497e8e8a9e3efc367de3 P d430585cc82548499c57a24dcd70308f: Deleting consensus metadata
I20260620 21:50:21.699368 3023 tablet_bootstrap.cc:492] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f: Bootstrap starting.
I20260620 21:50:21.703850 3023 tablet_bootstrap.cc:654] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:21.705541 3022 tablet_bootstrap.cc:492] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe: Bootstrap starting.
I20260620 21:50:21.706859 3022 tablet_bootstrap.cc:654] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:21.708076 3023 tablet_bootstrap.cc:492] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f: No bootstrap required, opened a new log
I20260620 21:50:21.708102 2500 catalog_manager.cc:4841] TS d430585cc82548499c57a24dcd70308f (127.0.193.194:36851): tablet 8b70a8b34166497e8e8a9e3efc367de3 (table test-table [id=56523b4f7e77455aaa898de70f1e0764]) successfully deleted
I20260620 21:50:21.708201 3023 ts_tablet_manager.cc:1397] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260620 21:50:21.708745 3023 raft_consensus.cc:348] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.708882 3023 raft_consensus.cc:374] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:21.708917 3023 raft_consensus.cc:729] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d430585cc82548499c57a24dcd70308f, State: Initialized, Role: FOLLOWER
I20260620 21:50:21.710005 3071 ts_tablet_manager.cc:1905] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:21.709301 3023 consensus_queue.cc:260] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.706877 3070 ts_tablet_manager.cc:1918] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:50:21.711364 3070 log.cc:1196] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/wal/wals/8b70a8b34166497e8e8a9e3efc367de3
I20260620 21:50:21.711738 3070 ts_tablet_manager.cc:1939] T 8b70a8b34166497e8e8a9e3efc367de3 P 65dfaf8db4194fbdb570567fa4eb3dfe: Deleting consensus metadata
I20260620 21:50:21.712030 3022 tablet_bootstrap.cc:492] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe: No bootstrap required, opened a new log
I20260620 21:50:21.712116 3022 ts_tablet_manager.cc:1397] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20260620 21:50:21.712639 3022 raft_consensus.cc:348] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.712787 3022 raft_consensus.cc:374] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:21.712833 3022 raft_consensus.cc:729] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65dfaf8db4194fbdb570567fa4eb3dfe, State: Initialized, Role: FOLLOWER
I20260620 21:50:21.713020 3022 consensus_queue.cc:260] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.713320 3022 ts_tablet_manager.cc:1428] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:50:21.714660 2502 catalog_manager.cc:4841] TS 65dfaf8db4194fbdb570567fa4eb3dfe (127.0.193.193:37635): tablet 8b70a8b34166497e8e8a9e3efc367de3 (table test-table [id=56523b4f7e77455aaa898de70f1e0764]) successfully deleted
I20260620 21:50:21.716185 3024 tablet_bootstrap.cc:492] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59: No bootstrap required, opened a new log
I20260620 21:50:21.716296 3024 ts_tablet_manager.cc:1397] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20260620 21:50:21.716840 3024 raft_consensus.cc:348] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.716959 3024 raft_consensus.cc:374] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:21.717005 3024 raft_consensus.cc:729] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5ccbdee09e481683ebfee375ad6f59, State: Initialized, Role: FOLLOWER
I20260620 21:50:21.717136 3024 consensus_queue.cc:260] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.717432 3024 ts_tablet_manager.cc:1428] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:50:21.717783 3071 ts_tablet_manager.cc:1918] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:50:21.717933 3071 log.cc:1196] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/8b70a8b34166497e8e8a9e3efc367de3
I20260620 21:50:21.718289 3071 ts_tablet_manager.cc:1939] T 8b70a8b34166497e8e8a9e3efc367de3 P 6e5ccbdee09e481683ebfee375ad6f59: Deleting consensus metadata
I20260620 21:50:21.719879 2501 catalog_manager.cc:4841] TS 6e5ccbdee09e481683ebfee375ad6f59 (127.0.193.195:40265): tablet 8b70a8b34166497e8e8a9e3efc367de3 (table test-table [id=56523b4f7e77455aaa898de70f1e0764]) successfully deleted
I20260620 21:50:21.721029 3023 ts_tablet_manager.cc:1428] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20260620 21:50:21.764487 3029 raft_consensus.cc:482] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:21.764652 3029 raft_consensus.cc:504] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.765130 3029 leader_election.cc:288] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65dfaf8db4194fbdb570567fa4eb3dfe (127.0.193.193:37635), d430585cc82548499c57a24dcd70308f (127.0.193.194:36851)
I20260620 21:50:21.765674 2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0082fffc76483fbc2b4187d1a37e27" candidate_uuid: "6e5ccbdee09e481683ebfee375ad6f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" is_pre_election: true
I20260620 21:50:21.765686 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0082fffc76483fbc2b4187d1a37e27" candidate_uuid: "6e5ccbdee09e481683ebfee375ad6f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d430585cc82548499c57a24dcd70308f" is_pre_election: true
I20260620 21:50:21.765816 2673 raft_consensus.cc:2454] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e5ccbdee09e481683ebfee375ad6f59 in term 0.
I20260620 21:50:21.765851 2813 raft_consensus.cc:2454] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e5ccbdee09e481683ebfee375ad6f59 in term 0.
I20260620 21:50:21.766278 2884 leader_election.cc:302] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e5ccbdee09e481683ebfee375ad6f59, d430585cc82548499c57a24dcd70308f; no voters:
I20260620 21:50:21.766467 3029 raft_consensus.cc:2781] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:21.766511 3029 raft_consensus.cc:482] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:21.766546 3029 raft_consensus.cc:3037] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:21.767783 3029 raft_consensus.cc:504] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.768141 3029 leader_election.cc:288] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [CANDIDATE]: Term 1 election: Requested vote from peers 65dfaf8db4194fbdb570567fa4eb3dfe (127.0.193.193:37635), d430585cc82548499c57a24dcd70308f (127.0.193.194:36851)
I20260620 21:50:21.768569 2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0082fffc76483fbc2b4187d1a37e27" candidate_uuid: "6e5ccbdee09e481683ebfee375ad6f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe"
I20260620 21:50:21.768675 2673 raft_consensus.cc:3037] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:21.769775 2673 raft_consensus.cc:2454] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e5ccbdee09e481683ebfee375ad6f59 in term 1.
I20260620 21:50:21.770031 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c0082fffc76483fbc2b4187d1a37e27" candidate_uuid: "6e5ccbdee09e481683ebfee375ad6f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d430585cc82548499c57a24dcd70308f"
I20260620 21:50:21.770150 2813 raft_consensus.cc:3037] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:21.770215 2886 leader_election.cc:302] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65dfaf8db4194fbdb570567fa4eb3dfe, 6e5ccbdee09e481683ebfee375ad6f59; no voters:
I20260620 21:50:21.770447 3029 raft_consensus.cc:2781] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:21.770525 3029 raft_consensus.cc:686] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 LEADER]: Becoming Leader. State: Replica: 6e5ccbdee09e481683ebfee375ad6f59, State: Running, Role: LEADER
I20260620 21:50:21.770663 3029 consensus_queue.cc:237] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:21.771412 2813 raft_consensus.cc:2454] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e5ccbdee09e481683ebfee375ad6f59 in term 1.
I20260620 21:50:21.772951 2515 catalog_manager.cc:5495] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e5ccbdee09e481683ebfee375ad6f59 (127.0.193.195). New cstate: current_term: 1 leader_uuid: "6e5ccbdee09e481683ebfee375ad6f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } health_report { overall_health: HEALTHY } } }
I20260620 21:50:21.807327 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.803468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.0.193.195:40265)
0620 21:50:21.803583 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.803587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.803631 (+ 44us) server_negotiation.cc:407] Connection header received
0620 21:50:21.803782 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.803784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.803825 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.803886 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:21.804156 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.804687 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.806103 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.806335 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.806429 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.806780 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:21.806872 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:21.806937 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.807081 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.807130 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:50:21.817305 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.814009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:40265 (local address 127.0.193.195:56619)
0620 21:50:21.814138 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:21.814155 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:21.814253 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:21.814486 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:21.814488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:21.814495 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:50:21.814605 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.814610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.815383 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.815390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.816124 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:21.816134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.816556 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.816559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:21.816705 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.816730 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:50:21.817011 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:50:21.817018 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:50:21.817084 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 21:50:21.817169 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":65}
I20260620 21:50:21.817305 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:21.814072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:56619 (local address 127.0.193.195:40265)
0620 21:50:21.814182 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:21.814189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:21.814203 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:50:21.814357 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:21.814360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:21.814401 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:21.814469 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:21.814699 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.815288 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.816234 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:21.816456 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:21.816606 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:21.816856 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:21.816928 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:21.817000 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:21.817084 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 21:50:21.817143 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":52}
I20260620 21:50:21.823971 2673 raft_consensus.cc:1261] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Refusing update from remote peer 6e5ccbdee09e481683ebfee375ad6f59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:21.824038 2813 raft_consensus.cc:1261] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Refusing update from remote peer 6e5ccbdee09e481683ebfee375ad6f59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:21.824496 3062 consensus_queue.cc:1035] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:21.824688 3029 consensus_queue.cc:1035] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:21.829083 3051 mvcc.cc:205] Tried to move back new op lower bound from 7299040140587712512 to 7299040140375789568. Current Snapshot: MvccSnapshot[applied={T|T < 7299040140587712512}]
I20260620 21:50:21.829414 3052 mvcc.cc:205] Tried to move back new op lower bound from 7299040140587712512 to 7299040140375789568. Current Snapshot: MvccSnapshot[applied={T|T < 7299040140587712512}]
W20260620 21:50:21.841521 3050 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 21:50:23.408167 2515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:23.408332 2515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:23.411080 2515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287: Sending DeleteTablet for 3 replicas of tablet 6c0082fffc76483fbc2b4187d1a37e27
I20260620 21:50:23.411857 2793 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0082fffc76483fbc2b4187d1a37e27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:23 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:33264
I20260620 21:50:23.411888 2653 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0082fffc76483fbc2b4187d1a37e27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:23 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:46714
I20260620 21:50:23.411888 2932 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c0082fffc76483fbc2b4187d1a37e27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:23 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:54402
I20260620 21:50:23.412297 3101 tablet_replica.cc:330] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59: stopping tablet replica
I20260620 21:50:23.412303 3100 tablet_replica.cc:330] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f: stopping tablet replica
I20260620 21:50:23.412317 3099 tablet_replica.cc:330] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe: stopping tablet replica
I20260620 21:50:23.412483 3101 raft_consensus.cc:2229] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:50:23.412483 3100 raft_consensus.cc:2229] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:23.412603 3099 raft_consensus.cc:2229] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:23.412638 3100 raft_consensus.cc:2258] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:23.412804 3099 raft_consensus.cc:2258] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:23.412879 3101 raft_consensus.cc:2258] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:23.413431 3100 ts_tablet_manager.cc:1905] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:23.413578 3101 ts_tablet_manager.cc:1905] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:23.413982 3099 ts_tablet_manager.cc:1905] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:23.416501 3100 ts_tablet_manager.cc:1918] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:50:23.416591 3101 ts_tablet_manager.cc:1918] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:50:23.416608 3100 log.cc:1196] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/6c0082fffc76483fbc2b4187d1a37e27
I20260620 21:50:23.416669 3101 log.cc:1196] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/6c0082fffc76483fbc2b4187d1a37e27
I20260620 21:50:23.416932 3100 ts_tablet_manager.cc:1939] T 6c0082fffc76483fbc2b4187d1a37e27 P d430585cc82548499c57a24dcd70308f: Deleting consensus metadata
I20260620 21:50:23.417049 3101 ts_tablet_manager.cc:1939] T 6c0082fffc76483fbc2b4187d1a37e27 P 6e5ccbdee09e481683ebfee375ad6f59: Deleting consensus metadata
I20260620 21:50:23.417639 3099 ts_tablet_manager.cc:1918] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:50:23.417759 3099 log.cc:1196] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/wal/wals/6c0082fffc76483fbc2b4187d1a37e27
I20260620 21:50:23.417883 2500 catalog_manager.cc:4841] TS d430585cc82548499c57a24dcd70308f (127.0.193.194:36851): tablet 6c0082fffc76483fbc2b4187d1a37e27 (table test-table [id=de413b095fae4118ab643f39bf036c4a]) successfully deleted
I20260620 21:50:23.417884 2501 catalog_manager.cc:4841] TS 6e5ccbdee09e481683ebfee375ad6f59 (127.0.193.195:40265): tablet 6c0082fffc76483fbc2b4187d1a37e27 (table test-table [id=de413b095fae4118ab643f39bf036c4a]) successfully deleted
I20260620 21:50:23.418031 3099 ts_tablet_manager.cc:1939] T 6c0082fffc76483fbc2b4187d1a37e27 P 65dfaf8db4194fbdb570567fa4eb3dfe: Deleting consensus metadata
I20260620 21:50:23.418844 2502 catalog_manager.cc:4841] TS 65dfaf8db4194fbdb570567fa4eb3dfe (127.0.193.193:37635): tablet 6c0082fffc76483fbc2b4187d1a37e27 (table test-table [id=de413b095fae4118ab643f39bf036c4a]) successfully deleted
Jun 20 21:50:23 dist-test-slave-kk5c krb5kdc[2471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:50:23 dist-test-slave-kk5c krb5kdc[2471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:23.449240 3109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:23.435233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.0.193.254:39551)
0620 21:50:23.435560 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:23.435567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:23.435584 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:50:23.435695 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:23.435697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:23.435739 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:23.435819 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:23.436600 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.437445 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.438393 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.438667 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.438757 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:23.445401 (+ 6644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:23.445405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:23.445408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:23.445446 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:23.447351 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:23.447933 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:23.447937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:23.447939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:23.448034 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:23.448313 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:23.448315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:23.448317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:23.448549 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:23.448671 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:23.448932 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 21:50:23.449062 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":139,"threads_started":1}
I20260620 21:50:23.452119 2515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:50:23.452608 2515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:23.457687 2793 tablet_service.cc:1467] Processing CreateTablet for tablet 432952686ee3419f8f4de2938da4bf89 (DEFAULT_TABLE table=test-table [id=52b7c215d8934363b354f5338e031b88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:23.457687 2653 tablet_service.cc:1467] Processing CreateTablet for tablet 432952686ee3419f8f4de2938da4bf89 (DEFAULT_TABLE table=test-table [id=52b7c215d8934363b354f5338e031b88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:23.457917 2932 tablet_service.cc:1467] Processing CreateTablet for tablet 432952686ee3419f8f4de2938da4bf89 (DEFAULT_TABLE table=test-table [id=52b7c215d8934363b354f5338e031b88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:23.458004 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432952686ee3419f8f4de2938da4bf89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:23.458004 2653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432952686ee3419f8f4de2938da4bf89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:23.458217 2932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432952686ee3419f8f4de2938da4bf89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:23.460778 3110 tablet_bootstrap.cc:492] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f: Bootstrap starting.
I20260620 21:50:23.460924 3112 tablet_bootstrap.cc:492] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59: Bootstrap starting.
I20260620 21:50:23.461674 3112 tablet_bootstrap.cc:654] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:23.461675 3110 tablet_bootstrap.cc:654] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:23.462605 3111 tablet_bootstrap.cc:492] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe: Bootstrap starting.
I20260620 21:50:23.462745 3112 tablet_bootstrap.cc:492] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59: No bootstrap required, opened a new log
I20260620 21:50:23.462831 3112 ts_tablet_manager.cc:1397] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:50:23.463274 3112 raft_consensus.cc:348] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.463419 3112 raft_consensus.cc:374] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:23.463461 3112 raft_consensus.cc:729] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5ccbdee09e481683ebfee375ad6f59, State: Initialized, Role: FOLLOWER
I20260620 21:50:23.463642 3111 tablet_bootstrap.cc:654] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:23.463609 3112 consensus_queue.cc:260] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.463999 3112 ts_tablet_manager.cc:1428] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:23.464566 3110 tablet_bootstrap.cc:492] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f: No bootstrap required, opened a new log
I20260620 21:50:23.464668 3110 ts_tablet_manager.cc:1397] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 21:50:23.464814 3111 tablet_bootstrap.cc:492] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe: No bootstrap required, opened a new log
I20260620 21:50:23.464916 3111 ts_tablet_manager.cc:1397] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:50:23.465186 3110 raft_consensus.cc:348] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.465304 3110 raft_consensus.cc:374] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:23.465335 3110 raft_consensus.cc:729] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d430585cc82548499c57a24dcd70308f, State: Initialized, Role: FOLLOWER
I20260620 21:50:23.465330 3111 raft_consensus.cc:348] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.465464 3110 consensus_queue.cc:260] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.465525 3111 raft_consensus.cc:374] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:23.465567 3111 raft_consensus.cc:729] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65dfaf8db4194fbdb570567fa4eb3dfe, State: Initialized, Role: FOLLOWER
I20260620 21:50:23.465704 3111 consensus_queue.cc:260] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.465848 3110 ts_tablet_manager.cc:1428] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 21:50:23.466064 3111 ts_tablet_manager.cc:1428] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:23.475308 3087 raft_consensus.cc:482] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:23.475443 3087 raft_consensus.cc:504] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.475869 3087 leader_election.cc:288] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65dfaf8db4194fbdb570567fa4eb3dfe (127.0.193.193:37635), d430585cc82548499c57a24dcd70308f (127.0.193.194:36851)
I20260620 21:50:23.476289 2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432952686ee3419f8f4de2938da4bf89" candidate_uuid: "6e5ccbdee09e481683ebfee375ad6f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" is_pre_election: true
I20260620 21:50:23.476305 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432952686ee3419f8f4de2938da4bf89" candidate_uuid: "6e5ccbdee09e481683ebfee375ad6f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d430585cc82548499c57a24dcd70308f" is_pre_election: true
I20260620 21:50:23.476442 2813 raft_consensus.cc:2454] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e5ccbdee09e481683ebfee375ad6f59 in term 0.
I20260620 21:50:23.476442 2673 raft_consensus.cc:2454] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e5ccbdee09e481683ebfee375ad6f59 in term 0.
I20260620 21:50:23.476778 2884 leader_election.cc:302] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e5ccbdee09e481683ebfee375ad6f59, d430585cc82548499c57a24dcd70308f; no voters:
I20260620 21:50:23.476931 3087 raft_consensus.cc:2781] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:23.476991 3087 raft_consensus.cc:482] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:23.477029 3087 raft_consensus.cc:3037] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:23.478042 3087 raft_consensus.cc:504] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.478372 3087 leader_election.cc:288] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [CANDIDATE]: Term 1 election: Requested vote from peers 65dfaf8db4194fbdb570567fa4eb3dfe (127.0.193.193:37635), d430585cc82548499c57a24dcd70308f (127.0.193.194:36851)
I20260620 21:50:23.478731 2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432952686ee3419f8f4de2938da4bf89" candidate_uuid: "6e5ccbdee09e481683ebfee375ad6f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe"
I20260620 21:50:23.478735 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432952686ee3419f8f4de2938da4bf89" candidate_uuid: "6e5ccbdee09e481683ebfee375ad6f59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d430585cc82548499c57a24dcd70308f"
I20260620 21:50:23.478873 2673 raft_consensus.cc:3037] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:23.478881 2813 raft_consensus.cc:3037] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:23.479866 2673 raft_consensus.cc:2454] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e5ccbdee09e481683ebfee375ad6f59 in term 1.
I20260620 21:50:23.479858 2813 raft_consensus.cc:2454] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e5ccbdee09e481683ebfee375ad6f59 in term 1.
I20260620 21:50:23.480288 2886 leader_election.cc:302] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65dfaf8db4194fbdb570567fa4eb3dfe, 6e5ccbdee09e481683ebfee375ad6f59; no voters:
I20260620 21:50:23.480540 3087 raft_consensus.cc:2781] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:23.480656 3087 raft_consensus.cc:686] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 LEADER]: Becoming Leader. State: Replica: 6e5ccbdee09e481683ebfee375ad6f59, State: Running, Role: LEADER
I20260620 21:50:23.480834 3087 consensus_queue.cc:237] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.482693 2516 catalog_manager.cc:5495] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e5ccbdee09e481683ebfee375ad6f59 (127.0.193.195). New cstate: current_term: 1 leader_uuid: "6e5ccbdee09e481683ebfee375ad6f59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } health_report { overall_health: HEALTHY } } }
I20260620 21:50:23.502419 3120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:23.499298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.0.193.195:40265)
0620 21:50:23.499546 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:23.499595 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:23.499608 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:50:23.499647 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:23.499649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:23.499690 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:23.499761 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:23.499987 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.500488 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.501281 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.501514 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.501625 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:23.501862 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:23.501931 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:23.501986 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:23.502076 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 21:50:23.502122 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":111,"threads_started":1}
I20260620 21:50:23.508963 2673 raft_consensus.cc:1261] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Refusing update from remote peer 6e5ccbdee09e481683ebfee375ad6f59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:23.509387 2813 raft_consensus.cc:1261] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Refusing update from remote peer 6e5ccbdee09e481683ebfee375ad6f59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:23.509497 3082 consensus_queue.cc:1035] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:23.509881 3082 consensus_queue.cc:1035] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:23.543757 2516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:23.543921 2516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:23.546295 2516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287: Sending DeleteTablet for 3 replicas of tablet 432952686ee3419f8f4de2938da4bf89
I20260620 21:50:23.547143 2653 tablet_service.cc:1514] Processing DeleteTablet for tablet 432952686ee3419f8f4de2938da4bf89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:23 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:46714
I20260620 21:50:23.547118 2793 tablet_service.cc:1514] Processing DeleteTablet for tablet 432952686ee3419f8f4de2938da4bf89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:23 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:33264
I20260620 21:50:23.547257 2932 tablet_service.cc:1514] Processing DeleteTablet for tablet 432952686ee3419f8f4de2938da4bf89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:23 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:54402
I20260620 21:50:23.547387 3099 tablet_replica.cc:330] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe: stopping tablet replica
I20260620 21:50:23.547456 3101 tablet_replica.cc:330] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59: stopping tablet replica
I20260620 21:50:23.547523 3099 raft_consensus.cc:2229] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:23.547556 3101 raft_consensus.cc:2229] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:50:23.547552 3100 tablet_replica.cc:330] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f: stopping tablet replica
I20260620 21:50:23.547657 3099 raft_consensus.cc:2258] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:23.547688 3100 raft_consensus.cc:2229] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:23.547744 3101 raft_consensus.cc:2258] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:23.547804 3100 raft_consensus.cc:2258] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:23.548444 3099 ts_tablet_manager.cc:1905] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:23.548444 3101 ts_tablet_manager.cc:1905] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:23.548727 3100 ts_tablet_manager.cc:1905] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:23.550969 3100 ts_tablet_manager.cc:1918] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:50:23.551025 3099 ts_tablet_manager.cc:1918] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:50:23.551025 3101 ts_tablet_manager.cc:1918] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:50:23.551065 3100 log.cc:1196] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/432952686ee3419f8f4de2938da4bf89
I20260620 21:50:23.551101 3101 log.cc:1196] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/432952686ee3419f8f4de2938da4bf89
I20260620 21:50:23.551101 3099 log.cc:1196] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-0/wal/wals/432952686ee3419f8f4de2938da4bf89
I20260620 21:50:23.550968 2516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 21:50:23.551381 3100 ts_tablet_manager.cc:1939] T 432952686ee3419f8f4de2938da4bf89 P d430585cc82548499c57a24dcd70308f: Deleting consensus metadata
I20260620 21:50:23.551406 3101 ts_tablet_manager.cc:1939] T 432952686ee3419f8f4de2938da4bf89 P 6e5ccbdee09e481683ebfee375ad6f59: Deleting consensus metadata
I20260620 21:50:23.551406 3099 ts_tablet_manager.cc:1939] T 432952686ee3419f8f4de2938da4bf89 P 65dfaf8db4194fbdb570567fa4eb3dfe: Deleting consensus metadata
W20260620 21:50:23.551697 2516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:23.552438 2500 catalog_manager.cc:4841] TS d430585cc82548499c57a24dcd70308f (127.0.193.194:36851): tablet 432952686ee3419f8f4de2938da4bf89 (table test-table [id=52b7c215d8934363b354f5338e031b88]) successfully deleted
I20260620 21:50:23.552507 2502 catalog_manager.cc:4841] TS 65dfaf8db4194fbdb570567fa4eb3dfe (127.0.193.193:37635): tablet 432952686ee3419f8f4de2938da4bf89 (table test-table [id=52b7c215d8934363b354f5338e031b88]) successfully deleted
I20260620 21:50:23.552716 2501 catalog_manager.cc:4841] TS 6e5ccbdee09e481683ebfee375ad6f59 (127.0.193.195:40265): tablet 432952686ee3419f8f4de2938da4bf89 (table test-table [id=52b7c215d8934363b354f5338e031b88]) successfully deleted
I20260620 21:50:23.557554 2653 tablet_service.cc:1467] Processing CreateTablet for tablet a46912719d574af49c1361656115c719 (DEFAULT_TABLE table=test-table [id=8811f3a6541c48e28387c05928019f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:23.557610 2793 tablet_service.cc:1467] Processing CreateTablet for tablet a46912719d574af49c1361656115c719 (DEFAULT_TABLE table=test-table [id=8811f3a6541c48e28387c05928019f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:23.557718 2932 tablet_service.cc:1467] Processing CreateTablet for tablet a46912719d574af49c1361656115c719 (DEFAULT_TABLE table=test-table [id=8811f3a6541c48e28387c05928019f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:23.557924 2653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46912719d574af49c1361656115c719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:23.557924 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46912719d574af49c1361656115c719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:23.558141 2932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46912719d574af49c1361656115c719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:23.560189 3112 tablet_bootstrap.cc:492] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59: Bootstrap starting.
I20260620 21:50:23.560534 3110 tablet_bootstrap.cc:492] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f: Bootstrap starting.
I20260620 21:50:23.560530 3111 tablet_bootstrap.cc:492] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe: Bootstrap starting.
I20260620 21:50:23.560945 3112 tablet_bootstrap.cc:654] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:23.561528 3111 tablet_bootstrap.cc:654] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:23.561548 3110 tablet_bootstrap.cc:654] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:23.561997 3112 tablet_bootstrap.cc:492] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59: No bootstrap required, opened a new log
I20260620 21:50:23.562100 3112 ts_tablet_manager.cc:1397] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:50:23.562516 3112 raft_consensus.cc:348] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.562635 3112 raft_consensus.cc:374] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:23.562666 3112 raft_consensus.cc:729] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5ccbdee09e481683ebfee375ad6f59, State: Initialized, Role: FOLLOWER
I20260620 21:50:23.562788 3112 consensus_queue.cc:260] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.562831 3111 tablet_bootstrap.cc:492] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe: No bootstrap required, opened a new log
I20260620 21:50:23.562925 3111 ts_tablet_manager.cc:1397] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:50:23.563053 3112 ts_tablet_manager.cc:1428] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:50:23.563359 3111 raft_consensus.cc:348] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.563480 3111 raft_consensus.cc:374] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:23.563521 3111 raft_consensus.cc:729] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65dfaf8db4194fbdb570567fa4eb3dfe, State: Initialized, Role: FOLLOWER
I20260620 21:50:23.563657 3111 consensus_queue.cc:260] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.563992 3111 ts_tablet_manager.cc:1428] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:23.564749 3110 tablet_bootstrap.cc:492] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f: No bootstrap required, opened a new log
I20260620 21:50:23.564829 3110 ts_tablet_manager.cc:1397] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 21:50:23.565182 3110 raft_consensus.cc:348] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.565274 3110 raft_consensus.cc:374] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:23.565299 3110 raft_consensus.cc:729] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d430585cc82548499c57a24dcd70308f, State: Initialized, Role: FOLLOWER
I20260620 21:50:23.565451 3110 consensus_queue.cc:260] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.565737 3110 ts_tablet_manager.cc:1428] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:23.696386 3116 raft_consensus.cc:482] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:23.696587 3116 raft_consensus.cc:504] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.697615 3116 leader_election.cc:288] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65dfaf8db4194fbdb570567fa4eb3dfe (127.0.193.193:37635), 6e5ccbdee09e481683ebfee375ad6f59 (127.0.193.195:40265)
I20260620 21:50:23.701551 3128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:23.697985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:51923 (local address 127.0.193.193:37635)
0620 21:50:23.698275 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:23.698288 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:23.698304 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:50:23.698463 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:23.698466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:23.698516 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:23.698607 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:23.698894 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.699498 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.700542 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.700952 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.701098 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:23.701157 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:23.701339 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 21:50:23.701413 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":106,"threads_started":1}
I20260620 21:50:23.701574 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:23.697888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:37635 (local address 127.0.193.194:51923)
0620 21:50:23.698200 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:23.698218 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:23.698339 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:23.698620 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:23.698623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:23.698663 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:23.698782 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:23.698787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.699611 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.699616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:23.700414 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:23.700422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.701094 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.701097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:23.701205 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:23.701232 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:50:23.701312 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 21:50:23.701386 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"spinlock_wait_cycles":24192,"thread_start_us":95,"threads_started":1}
I20260620 21:50:23.701741 3120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:23.698164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:58391 (local address 127.0.193.195:40265)
0620 21:50:23.698282 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:23.698287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:23.698618 (+ 331us) server_negotiation.cc:407] Connection header received
0620 21:50:23.698784 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:23.698787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:23.698835 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:23.698913 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:23.699167 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.699757 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.700758 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.701183 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.701308 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:23.701359 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:23.701537 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 21:50:23.701619 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 21:50:23.701875 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:23.697888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:40265 (local address 127.0.193.194:58391)
0620 21:50:23.698524 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:23.698536 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:23.698679 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:23.698925 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:23.698928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:23.698942 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:23.699067 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:23.699073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.699886 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.699889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:23.700642 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:23.700650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.701299 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.701303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:23.701404 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:23.701428 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:50:23.701520 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 21:50:23.701588 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":323,"spinlock_wait_cycles":768,"thread_start_us":123,"threads_started":1}
I20260620 21:50:23.702090 2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46912719d574af49c1361656115c719" candidate_uuid: "d430585cc82548499c57a24dcd70308f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" is_pre_election: true
I20260620 21:50:23.702248 2673 raft_consensus.cc:2454] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d430585cc82548499c57a24dcd70308f in term 0.
I20260620 21:50:23.702385 2953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46912719d574af49c1361656115c719" candidate_uuid: "d430585cc82548499c57a24dcd70308f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5ccbdee09e481683ebfee375ad6f59" is_pre_election: true
I20260620 21:50:23.702533 2953 raft_consensus.cc:2454] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d430585cc82548499c57a24dcd70308f in term 0.
I20260620 21:50:23.702647 2746 leader_election.cc:302] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65dfaf8db4194fbdb570567fa4eb3dfe, d430585cc82548499c57a24dcd70308f; no voters:
I20260620 21:50:23.702931 3116 raft_consensus.cc:2781] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:23.702986 3116 raft_consensus.cc:482] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:23.703013 3116 raft_consensus.cc:3037] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:23.704073 3116 raft_consensus.cc:504] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.704437 3116 leader_election.cc:288] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [CANDIDATE]: Term 1 election: Requested vote from peers 65dfaf8db4194fbdb570567fa4eb3dfe (127.0.193.193:37635), 6e5ccbdee09e481683ebfee375ad6f59 (127.0.193.195:40265)
I20260620 21:50:23.704757 2953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46912719d574af49c1361656115c719" candidate_uuid: "d430585cc82548499c57a24dcd70308f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5ccbdee09e481683ebfee375ad6f59"
I20260620 21:50:23.704818 2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46912719d574af49c1361656115c719" candidate_uuid: "d430585cc82548499c57a24dcd70308f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe"
I20260620 21:50:23.704870 2953 raft_consensus.cc:3037] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:23.704964 2673 raft_consensus.cc:3037] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:23.705775 2953 raft_consensus.cc:2454] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d430585cc82548499c57a24dcd70308f in term 1.
I20260620 21:50:23.705897 2673 raft_consensus.cc:2454] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d430585cc82548499c57a24dcd70308f in term 1.
I20260620 21:50:23.706048 2745 leader_election.cc:302] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e5ccbdee09e481683ebfee375ad6f59, d430585cc82548499c57a24dcd70308f; no voters:
I20260620 21:50:23.706233 3116 raft_consensus.cc:2781] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:23.706471 3116 raft_consensus.cc:686] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [term 1 LEADER]: Becoming Leader. State: Replica: d430585cc82548499c57a24dcd70308f, State: Running, Role: LEADER
I20260620 21:50:23.706717 3116 consensus_queue.cc:237] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } }
I20260620 21:50:23.709331 2514 catalog_manager.cc:5495] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f reported cstate change: term changed from 0 to 1, leader changed from <none> to d430585cc82548499c57a24dcd70308f (127.0.193.194). New cstate: current_term: 1 leader_uuid: "d430585cc82548499c57a24dcd70308f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d430585cc82548499c57a24dcd70308f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 } health_report { overall_health: UNKNOWN } } }
I20260620 21:50:23.731595 3132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:23.727762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.0.193.194:36851)
0620 21:50:23.728101 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:23.728110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:23.728129 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:50:23.728199 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:23.728201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:23.728252 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:23.728383 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:23.728609 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.729182 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.730051 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.730302 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.730403 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:23.730668 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:23.731130 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:23.731287 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:23.731349 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 21:50:23.731406 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":145,"threads_started":1}
I20260620 21:50:23.743435 3120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:23.739507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:39721 (local address 127.0.193.195:40265)
0620 21:50:23.739671 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:23.739675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:23.739845 (+ 170us) server_negotiation.cc:407] Connection header received
0620 21:50:23.739955 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:23.739957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:23.740007 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:23.740083 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:23.740391 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.740934 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.742203 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.742439 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.742528 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:23.742972 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:23.743045 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:23.743103 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:23.743249 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 21:50:23.743315 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:50:23.743460 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:23.739387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:40265 (local address 127.0.193.194:39721)
0620 21:50:23.739766 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:23.739779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:23.739875 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:23.740101 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:23.740104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:23.740110 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:50:23.740282 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:23.740288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.741044 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.741047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:23.742074 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:23.742084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.742712 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.742715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:23.742788 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:23.742835 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:50:23.743116 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:50:23.743121 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:50:23.743181 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 21:50:23.743323 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":176,"threads_started":1}
I20260620 21:50:23.749785 2953 raft_consensus.cc:1261] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Refusing update from remote peer d430585cc82548499c57a24dcd70308f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:23.749843 2673 raft_consensus.cc:1261] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Refusing update from remote peer d430585cc82548499c57a24dcd70308f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:23.750317 3130 consensus_queue.cc:1035] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65dfaf8db4194fbdb570567fa4eb3dfe" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:23.750509 3116 consensus_queue.cc:1035] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e5ccbdee09e481683ebfee375ad6f59" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 40265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:23.772728 3132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:23.768699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:44385 (local address 127.0.193.194:36851)
0620 21:50:23.768851 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:23.768855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:23.768873 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:50:23.768934 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:23.768940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:23.768995 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:23.769095 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:23.769401 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.770177 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.771538 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.771848 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.771964 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:23.772263 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:23.772347 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:23.772432 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:23.772523 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 21:50:23.772589 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:50:23.772740 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:23.763928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:36851 (local address 127.0.193.195:44385)
0620 21:50:23.764202 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:23.764219 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:23.764315 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:23.769150 (+ 4835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:23.769153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:23.769165 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:50:23.769286 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:23.769292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.770289 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.770293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:23.771366 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:23.771377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:23.771969 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:23.771973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:23.772075 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:23.772121 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:50:23.772435 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:50:23.772444 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 21:50:23.772512 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 21:50:23.772604 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":93,"threads_started":1}
W20260620 21:50:23.773702 3084 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 21:50:25.336163 2516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:25.336373 2516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:25.339143 2516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7119608480e745a9bd1b9328149af287: Sending DeleteTablet for 3 replicas of tablet a46912719d574af49c1361656115c719
I20260620 21:50:25.339879 2653 tablet_service.cc:1514] Processing DeleteTablet for tablet a46912719d574af49c1361656115c719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:25 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:46714
I20260620 21:50:25.340075 2932 tablet_service.cc:1514] Processing DeleteTablet for tablet a46912719d574af49c1361656115c719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:25 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:54402
I20260620 21:50:25.340229 3165 tablet_replica.cc:330] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe: stopping tablet replica
I20260620 21:50:25.340206 2793 tablet_service.cc:1514] Processing DeleteTablet for tablet a46912719d574af49c1361656115c719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:25 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:33264
I20260620 21:50:25.340361 3165 raft_consensus.cc:2229] T a46912719d574af49c1361656115c719 P 65dfaf8db4194fbdb570567fa4eb3dfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:25.340443 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 2588
I20260620 21:50:25.343380 3167 tablet_replica.cc:330] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f: stopping tablet replica
I20260620 21:50:25.343569 3167 raft_consensus.cc:2229] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:50:25.343637 3166 tablet_replica.cc:330] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59: stopping tablet replica
I20260620 21:50:25.343781 3166 raft_consensus.cc:2229] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:25.343904 3166 raft_consensus.cc:2258] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:25.343936 3167 raft_consensus.cc:2258] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:25.344662 3167 ts_tablet_manager.cc:1905] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:25.344725 3166 ts_tablet_manager.cc:1905] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:25.346902 3167 ts_tablet_manager.cc:1918] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:50:25.347000 3167 log.cc:1196] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/a46912719d574af49c1361656115c719
I20260620 21:50:25.347362 3167 ts_tablet_manager.cc:1939] T a46912719d574af49c1361656115c719 P d430585cc82548499c57a24dcd70308f: Deleting consensus metadata
I20260620 21:50:25.348145 2500 catalog_manager.cc:4841] TS d430585cc82548499c57a24dcd70308f (127.0.193.194:36851): tablet a46912719d574af49c1361656115c719 (table test-table [id=8811f3a6541c48e28387c05928019f01]) successfully deleted
I20260620 21:50:25.348204 3166 ts_tablet_manager.cc:1918] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 21:50:25.348280 3166 log.cc:1196] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/a46912719d574af49c1361656115c719
I20260620 21:50:25.348575 3166 ts_tablet_manager.cc:1939] T a46912719d574af49c1361656115c719 P 6e5ccbdee09e481683ebfee375ad6f59: Deleting consensus metadata
I20260620 21:50:25.349370 2501 catalog_manager.cc:4841] TS 6e5ccbdee09e481683ebfee375ad6f59 (127.0.193.195:40265): tablet a46912719d574af49c1361656115c719 (table test-table [id=8811f3a6541c48e28387c05928019f01]) successfully deleted
I20260620 21:50:25.349823 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 2728
W20260620 21:50:25.349938 2502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:37635 (error 108)
I20260620 21:50:25.351624 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:25.351240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:37635 (local address 127.0.0.1:38634)
0620 21:50:25.351467 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:25.351541 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
W20260620 21:50:25.351850 2502 catalog_manager.cc:4584] TS 65dfaf8db4194fbdb570567fa4eb3dfe (127.0.193.193:37635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a46912719d574af49c1361656115c719: Network error: Client connection negotiation failed: client connection to 127.0.193.193:37635: connect: Connection refused (error 111)
I20260620 21:50:25.357173 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 2868
I20260620 21:50:25.364485 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 2484
2026-06-20T21:50:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6311 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:50:25 dist-test-slave-kk5c krb5kdc[3172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:50:25 dist-test-slave-kk5c krb5kdc[3172](info): set up 2 sockets
Jun 20 21:50:25 dist-test-slave-kk5c krb5kdc[3172](info): commencing operation
krb5kdc: starting...
W20260620 21:50:26.448252 775 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:50:26 dist-test-slave-kk5c krb5kdc[3172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:50:26.579833 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:46265
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:46265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:26.717840 3185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:26.718238 3185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:26.718353 3185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:26.722599 3185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:26.722713 3185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:26.722749 3185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:26.722786 3185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:26.722836 3185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:26.728637 3185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:46265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:46265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:26.730083 3185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:26.731222 3185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:26.740422 3190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:26.740408 3191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:26.740566 3193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:26.740552 3185 server_base.cc:1034] running on GCE node
I20260620 21:50:26.741443 3185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:26.742574 3185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:26.743752 3185 hybrid_clock.cc:648] HybridClock initialized: now 1781992226743733 us; error 32 us; skew 500 ppm
Jun 20 21:50:26 dist-test-slave-kk5c krb5kdc[3172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:26.746958 3185 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:50:26.747977 3185 webserver.cc:460] Webserver started at http://127.0.193.254:45259/ using document root <none> and password file <none>
I20260620 21:50:26.748625 3185 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:26.748725 3185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:26.748975 3185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:26.751400 3185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "baaa17fc5c1c483ebd44349172de8340"
format_stamp: "Formatted at 2026-06-20 21:50:26 on dist-test-slave-kk5c"
server_key: "f07567e6a949e06ff146dcb5ef41be03"
server_key_iv: "119d61d3dff735f425b9001775925de7"
server_key_version: "encryptionkey@0"
I20260620 21:50:26.752039 3185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "baaa17fc5c1c483ebd44349172de8340"
format_stamp: "Formatted at 2026-06-20 21:50:26 on dist-test-slave-kk5c"
server_key: "f07567e6a949e06ff146dcb5ef41be03"
server_key_iv: "119d61d3dff735f425b9001775925de7"
server_key_version: "encryptionkey@0"
I20260620 21:50:26.756184 3185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:50:26.758944 3200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:26.760385 3185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:26.760617 3185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "baaa17fc5c1c483ebd44349172de8340"
format_stamp: "Formatted at 2026-06-20 21:50:26 on dist-test-slave-kk5c"
server_key: "f07567e6a949e06ff146dcb5ef41be03"
server_key_iv: "119d61d3dff735f425b9001775925de7"
server_key_version: "encryptionkey@0"
I20260620 21:50:26.760759 3185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:26.775041 3185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:26.779047 3185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:26.779326 3185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:26.786963 3185 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:46265
I20260620 21:50:26.786993 3252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:46265 every 8 connection(s)
I20260620 21:50:26.788206 3185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:26.791131 3253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:26.797358 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 3185
I20260620 21:50:26.797524 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:26.797791 775 external_mini_cluster.cc:1428] Setting key da5f4dcc8363ca45db6cf69fc56b9429
I20260620 21:50:26.797780 3253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340: Bootstrap starting.
I20260620 21:50:26.800516 3253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:26.801299 3253 log.cc:824] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:26.803508 3253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340: No bootstrap required, opened a new log
I20260620 21:50:26.807355 3253 raft_consensus.cc:348] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baaa17fc5c1c483ebd44349172de8340" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46265 } }
Jun 20 21:50:26 dist-test-slave-kk5c krb5kdc[3172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:26.807674 3253 raft_consensus.cc:374] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:26.807756 3253 raft_consensus.cc:729] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baaa17fc5c1c483ebd44349172de8340, State: Initialized, Role: FOLLOWER
I20260620 21:50:26.808290 3253 consensus_queue.cc:260] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baaa17fc5c1c483ebd44349172de8340" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46265 } }
I20260620 21:50:26.808423 3253 raft_consensus.cc:388] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:26.808521 3253 raft_consensus.cc:482] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:26.808624 3253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:26.809537 3253 raft_consensus.cc:504] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baaa17fc5c1c483ebd44349172de8340" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46265 } }
I20260620 21:50:26.809882 3253 leader_election.cc:302] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baaa17fc5c1c483ebd44349172de8340; no voters:
I20260620 21:50:26.810214 3253 leader_election.cc:288] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:26.810343 3258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:26.810595 3258 raft_consensus.cc:686] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [term 1 LEADER]: Becoming Leader. State: Replica: baaa17fc5c1c483ebd44349172de8340, State: Running, Role: LEADER
I20260620 21:50:26.810982 3258 consensus_queue.cc:237] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baaa17fc5c1c483ebd44349172de8340" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46265 } }
I20260620 21:50:26.811561 3253 sys_catalog.cc:564] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:26.812888 3260 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baaa17fc5c1c483ebd44349172de8340. Latest consensus state: current_term: 1 leader_uuid: "baaa17fc5c1c483ebd44349172de8340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baaa17fc5c1c483ebd44349172de8340" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46265 } } }
I20260620 21:50:26.813019 3260 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:26.813336 3259 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baaa17fc5c1c483ebd44349172de8340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baaa17fc5c1c483ebd44349172de8340" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46265 } } }
I20260620 21:50:26.813439 3259 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:26.815398 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:26.799360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.0.193.254:46265)
0620 21:50:26.799890 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:26.799919 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:26.799959 (+ 40us) server_negotiation.cc:407] Connection header received
0620 21:50:26.800748 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:26.800770 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:26.801158 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:26.801557 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:26.802380 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:26.803597 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:26.804575 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:26.805112 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:26.805305 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:26.808325 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:26.808330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:26.808345 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:26.808377 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:26.810756 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:26.811421 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:26.811427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:26.811433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:26.811618 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:26.811995 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:26.811998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:26.812000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:26.812565 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:26.813611 (+ 1046us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:26.813787 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 21:50:26.814325 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":194,"threads_started":1}
W20260620 21:50:26.816732 3273 catalog_manager.cc:1558] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:50:26.816828 3273 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:50:26.816884 3268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:26.817781 3268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:26.823833 3268 catalog_manager.cc:1347] Generated new cluster ID: 4b6fe1f7617a4982b61589931db1571e
I20260620 21:50:26.823921 3268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:26.876757 3268 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:26.878551 3268 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:26.893767 3268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baaa17fc5c1c483ebd44349172de8340: Generated new TSK 0
I20260620 21:50:26.895124 3268 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:50:26.932406 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46265
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:27.048879 3279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:27.049154 3279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:27.049197 3279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:27.053473 3279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:27.053548 3279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:27.053610 3279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:27.059156 3279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:27.060321 3279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:27.061386 3279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:27.071861 3287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:27.071818 3284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:27.071818 3285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:27.072108 3279 server_base.cc:1034] running on GCE node
I20260620 21:50:27.072780 3279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:27.073521 3279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:27.074738 3279 hybrid_clock.cc:648] HybridClock initialized: now 1781992227074690 us; error 29 us; skew 500 ppm
Jun 20 21:50:27 dist-test-slave-kk5c krb5kdc[3172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:27.077908 3279 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:50:27.078884 3279 webserver.cc:460] Webserver started at http://127.0.193.193:36683/ using document root <none> and password file <none>
I20260620 21:50:27.079599 3279 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:27.079669 3279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:27.079886 3279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:27.081993 3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "6854b87cc8fb4ed790b670139b0d3edf"
format_stamp: "Formatted at 2026-06-20 21:50:27 on dist-test-slave-kk5c"
server_key: "445cafb90de1c4c51468469aba275641"
server_key_iv: "c7b2223f0fd96c8f3e675fcd157410a9"
server_key_version: "encryptionkey@0"
I20260620 21:50:27.082562 3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "6854b87cc8fb4ed790b670139b0d3edf"
format_stamp: "Formatted at 2026-06-20 21:50:27 on dist-test-slave-kk5c"
server_key: "445cafb90de1c4c51468469aba275641"
server_key_iv: "c7b2223f0fd96c8f3e675fcd157410a9"
server_key_version: "encryptionkey@0"
I20260620 21:50:27.086380 3279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 21:50:27.088986 3294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:27.090281 3279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:50:27.090441 3279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "6854b87cc8fb4ed790b670139b0d3edf"
format_stamp: "Formatted at 2026-06-20 21:50:27 on dist-test-slave-kk5c"
server_key: "445cafb90de1c4c51468469aba275641"
server_key_iv: "c7b2223f0fd96c8f3e675fcd157410a9"
server_key_version: "encryptionkey@0"
I20260620 21:50:27.090579 3279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:27.109159 3279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:27.112908 3279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:27.113148 3279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:27.113811 3279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:27.114652 3279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:27.114724 3279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:27.114795 3279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:27.114840 3279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:27.124905 3279 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:37387
I20260620 21:50:27.124923 3407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:37387 every 8 connection(s)
I20260620 21:50:27.125864 3279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:27.129700 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 3279
I20260620 21:50:27.129787 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:50:27.130061 775 external_mini_cluster.cc:1428] Setting key 6e76859327cbeeef3e426cb0900d7c6b
Jun 20 21:50:27 dist-test-slave-kk5c krb5kdc[3172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
I20260620 21:50:27.141469 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:27.127855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:42535 (local address 127.0.193.254:46265)
0620 21:50:27.128001 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:27.128005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:27.128822 (+ 817us) server_negotiation.cc:407] Connection header received
0620 21:50:27.130308 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:27.130312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:27.130373 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:27.130477 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:27.132180 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:27.132762 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:27.134144 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:27.134390 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:27.134504 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:27.137334 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:27.137340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:27.137343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:27.137374 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:27.139224 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:27.139831 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:27.139835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:27.139837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:27.139930 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:27.140335 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:27.140338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:27.140339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:27.140533 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:27.140670 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:27.141209 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 21:50:27.141327 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
I20260620 21:50:27.142545 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:27.128170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:46265 (local address 127.0.193.193:42535)
0620 21:50:27.128653 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:27.128702 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:27.130053 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:27.130787 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:27.130799 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:27.131300 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:27.132011 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:27.132033 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:27.132907 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:27.132914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:27.134002 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:27.134012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:27.134552 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:27.134556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:27.134729 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:27.135364 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:27.135381 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:27.137150 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:27.139346 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:27.139351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:27.139365 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:27.139697 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:27.140033 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:27.140035 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:27.140037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:27.140196 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:27.140688 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:27.140698 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:27.140983 (+ 285us) client_negotiation.cc:769] Sending connection context
0620 21:50:27.141159 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 21:50:27.141480 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":108,"threads_started":1}
I20260620 21:50:27.143882 3408 heartbeater.cc:347] Connected to a master server at 127.0.193.254:46265
I20260620 21:50:27.144131 3408 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:27.144640 3408 heartbeater.cc:511] Master 127.0.193.254:46265 requested a full tablet report, sending...
I20260620 21:50:27.146174 3216 ts_manager.cc:194] Registered new tserver with Master: 6854b87cc8fb4ed790b670139b0d3edf (127.0.193.193:37387)
I20260620 21:50:27.147403 3216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:42535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:50:27.159528 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46265
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:27.274492 3413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:27.274844 3413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:27.274924 3413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:27.279139 3413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:27.279233 3413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:27.279297 3413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:27.284742 3413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:27.285889 3413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:27.286947 3413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:27.297099 3418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:27.297103 3419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:27.297103 3421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:27.297402 3413 server_base.cc:1034] running on GCE node
I20260620 21:50:27.297830 3413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:27.298537 3413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:27.299748 3413 hybrid_clock.cc:648] HybridClock initialized: now 1781992227299711 us; error 50 us; skew 500 ppm
Jun 20 21:50:27 dist-test-slave-kk5c krb5kdc[3172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:27.302961 3413 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:50:27.304057 3413 webserver.cc:460] Webserver started at http://127.0.193.194:41329/ using document root <none> and password file <none>
I20260620 21:50:27.304687 3413 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:27.304783 3413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:27.305038 3413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:27.307457 3413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "1259ebbed5cd42969995559c4836ead4"
format_stamp: "Formatted at 2026-06-20 21:50:27 on dist-test-slave-kk5c"
server_key: "1795f21ac46cece6b86777370d0468ba"
server_key_iv: "120c3a08959d725ebb8b0abad4f43cbe"
server_key_version: "encryptionkey@0"
I20260620 21:50:27.308111 3413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "1259ebbed5cd42969995559c4836ead4"
format_stamp: "Formatted at 2026-06-20 21:50:27 on dist-test-slave-kk5c"
server_key: "1795f21ac46cece6b86777370d0468ba"
server_key_iv: "120c3a08959d725ebb8b0abad4f43cbe"
server_key_version: "encryptionkey@0"
I20260620 21:50:27.312361 3413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:50:27.315480 3428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:27.316964 3413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:50:27.317214 3413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "1259ebbed5cd42969995559c4836ead4"
format_stamp: "Formatted at 2026-06-20 21:50:27 on dist-test-slave-kk5c"
server_key: "1795f21ac46cece6b86777370d0468ba"
server_key_iv: "120c3a08959d725ebb8b0abad4f43cbe"
server_key_version: "encryptionkey@0"
I20260620 21:50:27.317349 3413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:27.333662 3413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:27.337369 3413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:27.337590 3413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:27.338214 3413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:27.339087 3413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:27.339160 3413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:27.339260 3413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:27.339304 3413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:27.348899 3413 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:37655
I20260620 21:50:27.348925 3541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:37655 every 8 connection(s)
I20260620 21:50:27.350092 3413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:27.356518 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 3413
I20260620 21:50:27.356614 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:50:27.356827 775 external_mini_cluster.cc:1428] Setting key 3dbfd830ee46c6cc924d5d1d272e4290
Jun 20 21:50:27 dist-test-slave-kk5c krb5kdc[3172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
I20260620 21:50:27.367987 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:27.352181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:35909 (local address 127.0.193.254:46265)
0620 21:50:27.352330 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:27.352334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:27.353272 (+ 938us) server_negotiation.cc:407] Connection header received
0620 21:50:27.354756 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:27.354760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:27.354818 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:27.354929 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:27.356382 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:27.357073 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:27.359425 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:27.359686 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:27.359789 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:27.363416 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:27.363419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:27.363421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:27.363448 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:27.365129 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:27.365950 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:27.365954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:27.365956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:27.366064 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:27.366564 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:27.366567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:27.366568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:27.366797 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:27.366907 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:27.367730 (+ 823us) server_negotiation.cc:299] Negotiation successful
0620 21:50:27.367850 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:50:27.368922 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:27.352564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:46265 (local address 127.0.193.194:35909)
0620 21:50:27.353102 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:27.353155 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:27.354519 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:27.355109 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:27.355119 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:27.355572 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:27.356100 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:27.356115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:27.357272 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:27.357275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:27.359274 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:27.359295 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:27.359865 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:27.359872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:27.360139 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:27.360974 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:27.361068 (+ 94us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:27.363164 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:27.365276 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:27.365285 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:27.365328 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:27.365801 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:27.366188 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:27.366191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:27.366193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:27.366407 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:27.366943 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:27.366952 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:27.367358 (+ 406us) client_negotiation.cc:769] Sending connection context
0620 21:50:27.367617 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 21:50:27.367965 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":101,"threads_started":1}
I20260620 21:50:27.370397 3542 heartbeater.cc:347] Connected to a master server at 127.0.193.254:46265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
I20260620 21:50:27.370857 3542 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:27.371565 3542 heartbeater.cc:511] Master 127.0.193.254:46265 requested a full tablet report, sending...
I20260620 21:50:27.373162 3216 ts_manager.cc:194] Registered new tserver with Master: 1259ebbed5cd42969995559c4836ead4 (127.0.193.194:37655)
I20260620 21:50:27.373908 3216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:35909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:50:27.388310 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46265
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:27.510334 3547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:27.510644 3547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:27.510694 3547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:27.515965 3547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:27.516052 3547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:27.516216 3547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:27.522049 3547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:27.523128 3547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:27.524216 3547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:27.534343 3555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:27.534646 3547 server_base.cc:1034] running on GCE node
W20260620 21:50:27.534399 3552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:27.534343 3553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:27.535351 3547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:27.536057 3547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:27.537241 3547 hybrid_clock.cc:648] HybridClock initialized: now 1781992227537222 us; error 31 us; skew 500 ppm
Jun 20 21:50:27 dist-test-slave-kk5c krb5kdc[3172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:27.540266 3547 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:50:27.541208 3547 webserver.cc:460] Webserver started at http://127.0.193.195:36673/ using document root <none> and password file <none>
I20260620 21:50:27.541790 3547 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:27.541882 3547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:27.542135 3547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:27.544337 3547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "4b5496e7cdfb4d409066a685995f6437"
format_stamp: "Formatted at 2026-06-20 21:50:27 on dist-test-slave-kk5c"
server_key: "da8993b3c649308eab0beef0b2bbd42b"
server_key_iv: "89f7a34e9dcb89f37909f0f96935acfd"
server_key_version: "encryptionkey@0"
I20260620 21:50:27.544917 3547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "4b5496e7cdfb4d409066a685995f6437"
format_stamp: "Formatted at 2026-06-20 21:50:27 on dist-test-slave-kk5c"
server_key: "da8993b3c649308eab0beef0b2bbd42b"
server_key_iv: "89f7a34e9dcb89f37909f0f96935acfd"
server_key_version: "encryptionkey@0"
I20260620 21:50:27.548769 3547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:27.551390 3562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:27.552780 3547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:27.552940 3547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "4b5496e7cdfb4d409066a685995f6437"
format_stamp: "Formatted at 2026-06-20 21:50:27 on dist-test-slave-kk5c"
server_key: "da8993b3c649308eab0beef0b2bbd42b"
server_key_iv: "89f7a34e9dcb89f37909f0f96935acfd"
server_key_version: "encryptionkey@0"
I20260620 21:50:27.553072 3547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:27.567456 3547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:27.570848 3547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:27.571076 3547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:27.571733 3547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:27.572571 3547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:27.572642 3547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:27.572714 3547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:27.572767 3547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:27.582162 3547 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:38331
I20260620 21:50:27.582252 3675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:38331 every 8 connection(s)
I20260620 21:50:27.583158 3547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:50:27.585659 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 3547
I20260620 21:50:27.585763 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:50:27.586146 775 external_mini_cluster.cc:1428] Setting key f0a3b999ec631aa48121c4da9891fe01
Jun 20 21:50:27 dist-test-slave-kk5c krb5kdc[3172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:27.598024 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:27.585209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:37617 (local address 127.0.193.254:46265)
0620 21:50:27.585408 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:27.585415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:27.586444 (+ 1029us) server_negotiation.cc:407] Connection header received
0620 21:50:27.588166 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:27.588171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:27.588224 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:27.588324 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:27.589568 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:27.590106 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:27.591318 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:27.591552 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:27.591667 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:27.594325 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:27.594329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:27.594332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:27.594362 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:27.595813 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:27.596461 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:27.596464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:27.596466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:27.596553 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:27.596909 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:27.596920 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:27.596921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:27.597110 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:27.597219 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:27.597724 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 21:50:27.597851 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 21:50:27.599042 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:27.585622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:46265 (local address 127.0.193.195:37617)
0620 21:50:27.586262 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:27.586323 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:27.587952 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:27.588547 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:27.588554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:27.588923 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:27.589410 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:27.589424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:27.590235 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:27.590239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:27.591134 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:27.591143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:27.591704 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:27.591708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:27.591883 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:27.592529 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:27.592546 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:27.594149 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:27.596006 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:27.596013 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:27.596030 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:27.596331 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:27.596655 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:27.596658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:27.596660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:27.596804 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:27.597215 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:27.597222 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:27.597484 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 21:50:27.597684 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 21:50:27.598020 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":192,"threads_started":1}
I20260620 21:50:27.600387 3676 heartbeater.cc:347] Connected to a master server at 127.0.193.254:46265
I20260620 21:50:27.600718 3676 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:27.601289 3676 heartbeater.cc:511] Master 127.0.193.254:46265 requested a full tablet report, sending...
I20260620 21:50:27.602336 3216 ts_manager.cc:194] Registered new tserver with Master: 4b5496e7cdfb4d409066a685995f6437 (127.0.193.195:38331)
I20260620 21:50:27.603010 3216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:37617
I20260620 21:50:27.613852 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 21:50:27.629387 3256 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:27.627765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60900 (local address 127.0.193.254:46265)
0620 21:50:27.627923 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:27.627927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:27.628085 (+ 158us) server_negotiation.cc:407] Connection header received
0620 21:50:27.628260 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:27.628264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:27.628317 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:27.628406 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:27.629231 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60900: BlockingRecv error: recv got EOF from 127.0.0.1:60900 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:50:27.630467 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 3279
I20260620 21:50:27.636055 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 3413
I20260620 21:50:27.642262 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 3547
I20260620 21:50:27.648635 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 3185
2026-06-20T21:50:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2284 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:50:27 dist-test-slave-kk5c krb5kdc[3688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:50:27 dist-test-slave-kk5c krb5kdc[3688](info): set up 2 sockets
Jun 20 21:50:27 dist-test-slave-kk5c krb5kdc[3688](info): commencing operation
krb5kdc: starting...
W20260620 21:50:28.723488 775 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:50:28 dist-test-slave-kk5c krb5kdc[3688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:50:28.858809 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:43417
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:38201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:43417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:28.977136 3701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:28.977488 3701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:28.977574 3701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:28.981945 3701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:28.982048 3701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:28.982087 3701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:28.982146 3701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:28.982183 3701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:28.987919 3701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:38201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:43417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:43417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:28.989130 3701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:28.990257 3701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:28.999646 3706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:28.999670 3709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:28.999677 3707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:28.999670 3701 server_base.cc:1034] running on GCE node
I20260620 21:50:29.000695 3701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:29.001963 3701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:29.003223 3701 hybrid_clock.cc:648] HybridClock initialized: now 1781992229003170 us; error 33 us; skew 500 ppm
Jun 20 21:50:29 dist-test-slave-kk5c krb5kdc[3688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:29.011178 3701 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:50:29.012374 3701 webserver.cc:460] Webserver started at http://127.0.193.254:42455/ using document root <none> and password file <none>
I20260620 21:50:29.012998 3701 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:29.013098 3701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:29.013358 3701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:29.015859 3701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "95d09c3362f241af9191ee2c2c13c78a"
format_stamp: "Formatted at 2026-06-20 21:50:29 on dist-test-slave-kk5c"
server_key: "7124430785def0dccb1e797fef93527c"
server_key_iv: "88e4636ec99f1c889afcf62b125fabf6"
server_key_version: "encryptionkey@0"
I20260620 21:50:29.016533 3701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "95d09c3362f241af9191ee2c2c13c78a"
format_stamp: "Formatted at 2026-06-20 21:50:29 on dist-test-slave-kk5c"
server_key: "7124430785def0dccb1e797fef93527c"
server_key_iv: "88e4636ec99f1c889afcf62b125fabf6"
server_key_version: "encryptionkey@0"
I20260620 21:50:29.020684 3701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:50:29.023487 3716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:29.025075 3701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:29.025368 3701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "95d09c3362f241af9191ee2c2c13c78a"
format_stamp: "Formatted at 2026-06-20 21:50:29 on dist-test-slave-kk5c"
server_key: "7124430785def0dccb1e797fef93527c"
server_key_iv: "88e4636ec99f1c889afcf62b125fabf6"
server_key_version: "encryptionkey@0"
I20260620 21:50:29.025550 3701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:29.036932 3701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:29.040504 3701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:29.040797 3701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:29.048359 3701 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:43417
I20260620 21:50:29.048702 3768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:43417 every 8 connection(s)
I20260620 21:50:29.049392 3701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:29.052368 3769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:29.056037 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 3701
I20260620 21:50:29.056221 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:29.056478 775 external_mini_cluster.cc:1428] Setting key 5b0e692daff4daf6e1345355c5b97856
I20260620 21:50:29.058674 3769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a: Bootstrap starting.
I20260620 21:50:29.061340 3769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:29.062232 3769 log.cc:824] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:29.064417 3769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a: No bootstrap required, opened a new log
Jun 20 21:50:29 dist-test-slave-kk5c krb5kdc[3688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:29.068674 3769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d09c3362f241af9191ee2c2c13c78a" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43417 } }
I20260620 21:50:29.068921 3769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:29.069044 3769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d09c3362f241af9191ee2c2c13c78a, State: Initialized, Role: FOLLOWER
I20260620 21:50:29.069653 3769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d09c3362f241af9191ee2c2c13c78a" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43417 } }
I20260620 21:50:29.069837 3769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:29.069911 3769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:29.070010 3769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:29.071498 3769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d09c3362f241af9191ee2c2c13c78a" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43417 } }
I20260620 21:50:29.071902 3769 leader_election.cc:302] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95d09c3362f241af9191ee2c2c13c78a; no voters:
I20260620 21:50:29.072156 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:29.057779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50786 (local address 127.0.193.254:43417)
0620 21:50:29.058192 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:29.058207 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:29.058249 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:50:29.058929 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:29.058947 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:29.059381 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:29.059675 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:29.060581 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:29.061707 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:29.062871 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:29.063431 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:29.063605 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:29.066553 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:29.066563 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:29.066583 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:29.066625 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:29.068498 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:29.069021 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:29.069029 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:29.069036 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:29.069206 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:29.069504 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:29.069507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:29.069508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:29.070094 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:29.070403 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:29.070648 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 21:50:29.071004 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":140,"threads_started":1}
I20260620 21:50:29.072281 3769 leader_election.cc:288] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:29.072448 3774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:29.072693 3774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [term 1 LEADER]: Becoming Leader. State: Replica: 95d09c3362f241af9191ee2c2c13c78a, State: Running, Role: LEADER
I20260620 21:50:29.073050 3774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d09c3362f241af9191ee2c2c13c78a" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43417 } }
I20260620 21:50:29.073675 3769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:29.074731 3775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95d09c3362f241af9191ee2c2c13c78a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d09c3362f241af9191ee2c2c13c78a" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43417 } } }
I20260620 21:50:29.074874 3775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:29.075264 3775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95d09c3362f241af9191ee2c2c13c78a. Latest consensus state: current_term: 1 leader_uuid: "95d09c3362f241af9191ee2c2c13c78a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d09c3362f241af9191ee2c2c13c78a" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43417 } } }
I20260620 21:50:29.075403 3775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:29.075766 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 21:50:29.079358 3789 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:50:29.079470 3789 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:50:29.079559 3783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:29.080397 3783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:29.087062 3783 catalog_manager.cc:1347] Generated new cluster ID: 20da3ef12324416f877a61f5f95154fc
I20260620 21:50:29.087158 3783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:29.102979 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:29.095986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.0.193.254:43417)
0620 21:50:29.096120 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:29.096126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:29.096416 (+ 290us) server_negotiation.cc:407] Connection header received
0620 21:50:29.096697 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:29.096701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:29.096760 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:29.096856 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:29.097537 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:29.098365 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:29.099640 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:29.100059 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:29.100172 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:29.101380 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:29.101398 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:29.101400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:29.102018 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:29.102062 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:29.102083 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:29.102395 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:29.102467 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 21:50:29.102575 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:50:29.110909 3783 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:29.112280 3783 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:29.124089 3783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95d09c3362f241af9191ee2c2c13c78a: Generated new TSK 0
I20260620 21:50:29.125254 3783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:50:29.136222 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 3701
2026-06-20T21:50:29Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1484 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:50:29 dist-test-slave-kk5c krb5kdc[3802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:50:29 dist-test-slave-kk5c krb5kdc[3802](info): set up 2 sockets
Jun 20 21:50:29 dist-test-slave-kk5c krb5kdc[3802](info): commencing operation
krb5kdc: starting...
W20260620 21:50:30.247473 775 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:50:30 dist-test-slave-kk5c krb5kdc[3802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:50:30.382992 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:44059
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:44059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:30.501863 3815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:30.502246 3815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:30.502338 3815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:30.506567 3815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:30.506687 3815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:30.506729 3815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:30.506768 3815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:30.506801 3815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:30.512369 3815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:44059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:30.513515 3815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:30.514617 3815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:30.523666 3820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:30.523692 3823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:30.523788 3815 server_base.cc:1034] running on GCE node
W20260620 21:50:30.523680 3821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:30.524583 3815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:30.525794 3815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:30.527017 3815 hybrid_clock.cc:648] HybridClock initialized: now 1781992230526986 us; error 47 us; skew 500 ppm
Jun 20 21:50:30 dist-test-slave-kk5c krb5kdc[3802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:30.530323 3815 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:50:30.531487 3815 webserver.cc:460] Webserver started at http://127.0.193.254:33145/ using document root <none> and password file <none>
I20260620 21:50:30.532116 3815 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:30.532217 3815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:30.532459 3815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:30.534651 3815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "9a014706e5f54a64aa66619907a9fff5"
format_stamp: "Formatted at 2026-06-20 21:50:30 on dist-test-slave-kk5c"
server_key: "7e0fe69509efb974f0944c1d914ec122"
server_key_iv: "6b8fe0f046b0bfbd27083b7a24f817d5"
server_key_version: "encryptionkey@0"
I20260620 21:50:30.535346 3815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "9a014706e5f54a64aa66619907a9fff5"
format_stamp: "Formatted at 2026-06-20 21:50:30 on dist-test-slave-kk5c"
server_key: "7e0fe69509efb974f0944c1d914ec122"
server_key_iv: "6b8fe0f046b0bfbd27083b7a24f817d5"
server_key_version: "encryptionkey@0"
I20260620 21:50:30.539414 3815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:50:30.542279 3830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:30.543979 3815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:30.544173 3815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "9a014706e5f54a64aa66619907a9fff5"
format_stamp: "Formatted at 2026-06-20 21:50:30 on dist-test-slave-kk5c"
server_key: "7e0fe69509efb974f0944c1d914ec122"
server_key_iv: "6b8fe0f046b0bfbd27083b7a24f817d5"
server_key_version: "encryptionkey@0"
I20260620 21:50:30.544320 3815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:30.585243 3815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:30.588428 3815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:30.588708 3815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:30.596362 3815 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:44059
I20260620 21:50:30.596607 3882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:44059 every 8 connection(s)
I20260620 21:50:30.597638 3815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:30.600644 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 3815
I20260620 21:50:30.600662 3883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:30.600811 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:30.601133 775 external_mini_cluster.cc:1428] Setting key 5425ccbf23c5935edabe6637bb64eb08
I20260620 21:50:30.608121 3883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5: Bootstrap starting.
Jun 20 21:50:30 dist-test-slave-kk5c krb5kdc[3802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:30.611135 3883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:30.612136 3883 log.cc:824] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:30.614666 3883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5: No bootstrap required, opened a new log
I20260620 21:50:30.617764 3886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:30.602620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.0.193.254:44059)
0620 21:50:30.603089 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:30.603107 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:30.603145 (+ 38us) server_negotiation.cc:407] Connection header received
0620 21:50:30.604014 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:30.604036 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:30.604388 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:30.604696 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:30.605691 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:30.606876 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:30.608144 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:30.608720 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:30.608861 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:30.611934 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:30.611940 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:30.611955 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:30.611988 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:30.614116 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:30.614710 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:30.614718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:30.614725 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:30.614875 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:30.615180 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:30.615183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:30.615208 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:30.615774 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:30.615983 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:30.616297 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 21:50:30.616636 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":153,"threads_started":1}
I20260620 21:50:30.619565 3883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a014706e5f54a64aa66619907a9fff5" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44059 } }
I20260620 21:50:30.619820 3883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:30.619856 3883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a014706e5f54a64aa66619907a9fff5, State: Initialized, Role: FOLLOWER
I20260620 21:50:30.620394 3883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a014706e5f54a64aa66619907a9fff5" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44059 } }
I20260620 21:50:30.620565 3883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:30.620683 3883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:30.620797 3883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:30.621820 3883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a014706e5f54a64aa66619907a9fff5" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44059 } }
I20260620 21:50:30.622213 3883 leader_election.cc:302] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a014706e5f54a64aa66619907a9fff5; no voters:
I20260620 21:50:30.622750 3883 leader_election.cc:288] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:30.622874 3888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:30.623168 3888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [term 1 LEADER]: Becoming Leader. State: Replica: 9a014706e5f54a64aa66619907a9fff5, State: Running, Role: LEADER
I20260620 21:50:30.623556 3888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a014706e5f54a64aa66619907a9fff5" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44059 } }
I20260620 21:50:30.624181 3883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:30.625471 3890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a014706e5f54a64aa66619907a9fff5. Latest consensus state: current_term: 1 leader_uuid: "9a014706e5f54a64aa66619907a9fff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a014706e5f54a64aa66619907a9fff5" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44059 } } }
I20260620 21:50:30.625654 3890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:30.625473 3889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a014706e5f54a64aa66619907a9fff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a014706e5f54a64aa66619907a9fff5" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44059 } } }
I20260620 21:50:30.625695 3889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5 [sys.catalog]: This master's current role is: LEADER
W20260620 21:50:30.629945 3903 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:50:30.630079 3903 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:50:30.630162 3897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:30.631323 3897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:30.638120 3897 catalog_manager.cc:1347] Generated new cluster ID: da01f1855a0a49d68546680ffa79e5c9
I20260620 21:50:30.638221 3897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:30.651638 3897 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:30.653308 3897 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:30.664114 3897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a014706e5f54a64aa66619907a9fff5: Generated new TSK 0
I20260620 21:50:30.665064 3897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:50:30.700215 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:44059
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:30.822793 3909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:30.823149 3909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:30.823271 3909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:30.827807 3909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:30.827916 3909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:30.828011 3909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:30.834295 3909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:30.835958 3909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:30.837181 3909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:30.849797 3915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:30.849797 3914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:30.849797 3917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:30.850394 3909 server_base.cc:1034] running on GCE node
I20260620 21:50:30.850885 3909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:30.851751 3909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:30.855171 3909 hybrid_clock.cc:648] HybridClock initialized: now 1781992230855130 us; error 44 us; skew 500 ppm
Jun 20 21:50:30 dist-test-slave-kk5c krb5kdc[3802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:30.858830 3909 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:50:30.860013 3909 webserver.cc:460] Webserver started at http://127.0.193.193:43131/ using document root <none> and password file <none>
I20260620 21:50:30.860706 3909 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:30.860780 3909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:30.861013 3909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:30.863576 3909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "d4d9cb159d8a4d4aab305a241252f08b"
format_stamp: "Formatted at 2026-06-20 21:50:30 on dist-test-slave-kk5c"
server_key: "b3b561d15ba76a08ccd6052f81ed94aa"
server_key_iv: "c849570c661944a635c8a356186d6a4e"
server_key_version: "encryptionkey@0"
I20260620 21:50:30.864244 3909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "d4d9cb159d8a4d4aab305a241252f08b"
format_stamp: "Formatted at 2026-06-20 21:50:30 on dist-test-slave-kk5c"
server_key: "b3b561d15ba76a08ccd6052f81ed94aa"
server_key_iv: "c849570c661944a635c8a356186d6a4e"
server_key_version: "encryptionkey@0"
I20260620 21:50:30.868856 3909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 21:50:30.872344 3924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:30.874012 3909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:50:30.874305 3909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "d4d9cb159d8a4d4aab305a241252f08b"
format_stamp: "Formatted at 2026-06-20 21:50:30 on dist-test-slave-kk5c"
server_key: "b3b561d15ba76a08ccd6052f81ed94aa"
server_key_iv: "c849570c661944a635c8a356186d6a4e"
server_key_version: "encryptionkey@0"
I20260620 21:50:30.874580 3909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:30.900219 3909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:30.904090 3909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:30.904320 3909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:30.904973 3909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:30.905850 3909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:30.905923 3909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:30.905997 3909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:30.906044 3909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:30.915774 3909 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:36827
I20260620 21:50:30.915863 4037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:36827 every 8 connection(s)
I20260620 21:50:30.916885 3909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:30.918381 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 3909
I20260620 21:50:30.918511 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:50:30.918814 775 external_mini_cluster.cc:1428] Setting key 999f4bfb718d4022e6fc2f05abc7be80
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
Jun 20 21:50:30 dist-test-slave-kk5c krb5kdc[3802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:30.941298 3886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:30.919254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:56845 (local address 127.0.193.254:44059)
0620 21:50:30.919403 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:30.919409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:30.921019 (+ 1610us) server_negotiation.cc:407] Connection header received
0620 21:50:30.923015 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:30.923019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:30.923080 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:30.923165 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:30.924547 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:30.925187 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:30.927159 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:30.927466 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:30.927559 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:30.935604 (+ 8045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:30.935608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:30.935610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:30.935677 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:30.937871 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:30.938693 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:30.938700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:30.938705 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:30.938825 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:30.939562 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:30.939567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:30.939569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:30.939864 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:30.940000 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:30.940682 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 21:50:30.940860 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:50:30.942504 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:30.919824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44059 (local address 127.0.193.193:56845)
0620 21:50:30.920812 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:30.920882 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:30.922765 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:30.923415 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:30.923423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:30.923815 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:30.924389 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:30.924405 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:30.925322 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:30.925328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:30.926992 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:30.927004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:30.927644 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:30.927650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:30.927886 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:30.928639 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:30.928665 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:30.935300 (+ 6635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:30.938185 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:30.938192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:30.938210 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:30.938545 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:30.939157 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:30.939161 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:30.939163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:30.939408 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:30.940038 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:30.940045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:30.940359 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 21:50:30.940536 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 21:50:30.941272 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":391,"threads_started":1}
I20260620 21:50:30.945047 4038 heartbeater.cc:347] Connected to a master server at 127.0.193.254:44059
I20260620 21:50:30.945363 4038 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:30.946013 4038 heartbeater.cc:511] Master 127.0.193.254:44059 requested a full tablet report, sending...
I20260620 21:50:30.947944 3847 ts_manager.cc:194] Registered new tserver with Master: d4d9cb159d8a4d4aab305a241252f08b (127.0.193.193:36827)
I20260620 21:50:30.949326 3847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:56845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:50:30.957299 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:44059
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:31.082773 4043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:31.083077 4043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:31.083123 4043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:31.087577 4043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:31.087664 4043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:31.087728 4043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:31.093266 4043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:31.094426 4043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:31.095453 4043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:31.106063 4051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:31.106022 4049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:31.106022 4048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:31.106422 4043 server_base.cc:1034] running on GCE node
I20260620 21:50:31.107005 4043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:31.107795 4043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:31.109007 4043 hybrid_clock.cc:648] HybridClock initialized: now 1781992231108991 us; error 34 us; skew 500 ppm
Jun 20 21:50:31 dist-test-slave-kk5c krb5kdc[3802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:31.112227 4043 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:50:31.113322 4043 webserver.cc:460] Webserver started at http://127.0.193.194:33599/ using document root <none> and password file <none>
I20260620 21:50:31.113942 4043 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:31.114041 4043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:31.114265 4043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:31.116497 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "36b801c05d734cc781c3ca0064cc69e5"
format_stamp: "Formatted at 2026-06-20 21:50:31 on dist-test-slave-kk5c"
server_key: "7ef1bade23e4d5ffc7982cf5f2ad7b57"
server_key_iv: "095101b0b2dd72f43475bb9c80052ed6"
server_key_version: "encryptionkey@0"
I20260620 21:50:31.117106 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "36b801c05d734cc781c3ca0064cc69e5"
format_stamp: "Formatted at 2026-06-20 21:50:31 on dist-test-slave-kk5c"
server_key: "7ef1bade23e4d5ffc7982cf5f2ad7b57"
server_key_iv: "095101b0b2dd72f43475bb9c80052ed6"
server_key_version: "encryptionkey@0"
I20260620 21:50:31.121137 4043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:31.124156 4058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:31.125756 4043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 21:50:31.125977 4043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "36b801c05d734cc781c3ca0064cc69e5"
format_stamp: "Formatted at 2026-06-20 21:50:31 on dist-test-slave-kk5c"
server_key: "7ef1bade23e4d5ffc7982cf5f2ad7b57"
server_key_iv: "095101b0b2dd72f43475bb9c80052ed6"
server_key_version: "encryptionkey@0"
I20260620 21:50:31.126125 4043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:31.149449 4043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:31.152989 4043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:31.153218 4043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:31.153855 4043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:31.154771 4043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:31.154847 4043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:31.154922 4043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:31.154964 4043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:31.164430 4043 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:35361
I20260620 21:50:31.164472 4171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:35361 every 8 connection(s)
I20260620 21:50:31.165412 4043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:31.175444 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 4043
I20260620 21:50:31.175590 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:50:31.175913 775 external_mini_cluster.cc:1428] Setting key 54db90f409ceffd5edb206dfd887517d
Jun 20 21:50:31 dist-test-slave-kk5c krb5kdc[3802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:31.182700 3886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:31.167379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:51225 (local address 127.0.193.254:44059)
0620 21:50:31.167513 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:31.167517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:31.168438 (+ 921us) server_negotiation.cc:407] Connection header received
0620 21:50:31.169884 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:31.169887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:31.169938 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:31.170019 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:31.171675 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:31.172336 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:31.173654 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:31.173887 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:31.173987 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:31.176780 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:31.176785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:31.176788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:31.176819 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:31.179227 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:31.179957 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:31.179960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:31.179962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:31.180046 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:31.180714 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:31.180717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:31.180718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:31.180922 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:31.181142 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:31.181923 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 21:50:31.182438 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:50:31.183413 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:31.167781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44059 (local address 127.0.193.194:51225)
0620 21:50:31.168232 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:31.168275 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:31.169658 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:31.170426 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:31.170434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:31.170849 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:31.171456 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:31.171523 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:31.172542 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:31.172545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:31.173513 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:31.173523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:31.174037 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:31.174040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:31.174185 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:31.174775 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:31.174792 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:31.176565 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:31.179385 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:31.179392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:31.179407 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:31.179801 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:31.180299 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:31.180304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:31.180307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:31.180547 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:31.181313 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:31.181320 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:31.181631 (+ 311us) client_negotiation.cc:769] Sending connection context
0620 21:50:31.181813 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 21:50:31.182148 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
I20260620 21:50:31.184890 4172 heartbeater.cc:347] Connected to a master server at 127.0.193.254:44059
I20260620 21:50:31.185178 4172 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:31.185711 4172 heartbeater.cc:511] Master 127.0.193.254:44059 requested a full tablet report, sending...
I20260620 21:50:31.187417 3847 ts_manager.cc:194] Registered new tserver with Master: 36b801c05d734cc781c3ca0064cc69e5 (127.0.193.194:35361)
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
I20260620 21:50:31.188236 3847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:51225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:50:31.206830 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:44059
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:31.338208 4177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:31.338550 4177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:31.338642 4177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:31.343328 4177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:31.343456 4177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:31.343565 4177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:31.350008 4177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:31.351280 4177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:31.352393 4177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:31.364614 4185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:31.364851 4182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:31.364614 4183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:31.365067 4177 server_base.cc:1034] running on GCE node
I20260620 21:50:31.365549 4177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:31.366292 4177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:31.367492 4177 hybrid_clock.cc:648] HybridClock initialized: now 1781992231367449 us; error 55 us; skew 500 ppm
Jun 20 21:50:31 dist-test-slave-kk5c krb5kdc[3802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:31.370846 4177 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:50:31.371979 4177 webserver.cc:460] Webserver started at http://127.0.193.195:40475/ using document root <none> and password file <none>
I20260620 21:50:31.372591 4177 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:31.372716 4177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:31.372975 4177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:31.375286 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "7fbd094319ef44d2b8e121e90602bfdd"
format_stamp: "Formatted at 2026-06-20 21:50:31 on dist-test-slave-kk5c"
server_key: "3ba7901ee59ecfd41a0d35e51d7f8e7b"
server_key_iv: "2c206007533312a55a97e4e324f34ea9"
server_key_version: "encryptionkey@0"
I20260620 21:50:31.375885 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "7fbd094319ef44d2b8e121e90602bfdd"
format_stamp: "Formatted at 2026-06-20 21:50:31 on dist-test-slave-kk5c"
server_key: "3ba7901ee59ecfd41a0d35e51d7f8e7b"
server_key_iv: "2c206007533312a55a97e4e324f34ea9"
server_key_version: "encryptionkey@0"
I20260620 21:50:31.380407 4177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:50:31.383481 4192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:31.385542 4177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:31.385677 4177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "7fbd094319ef44d2b8e121e90602bfdd"
format_stamp: "Formatted at 2026-06-20 21:50:31 on dist-test-slave-kk5c"
server_key: "3ba7901ee59ecfd41a0d35e51d7f8e7b"
server_key_iv: "2c206007533312a55a97e4e324f34ea9"
server_key_version: "encryptionkey@0"
I20260620 21:50:31.385792 4177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:31.406739 4177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:31.410271 4177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:31.410523 4177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:31.411099 4177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:31.411983 4177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:31.412032 4177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:31.412073 4177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:31.412088 4177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:31.421440 4177 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:35181
I20260620 21:50:31.421475 4305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:35181 every 8 connection(s)
I20260620 21:50:31.422402 4177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:50:31.425302 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 4177
I20260620 21:50:31.425448 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:50:31.425796 775 external_mini_cluster.cc:1428] Setting key 118dba34cfb4e5fe30271fcf3755a451
Jun 20 21:50:31 dist-test-slave-kk5c krb5kdc[3802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:31.437505 3886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:31.424468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:52289 (local address 127.0.193.254:44059)
0620 21:50:31.424659 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:31.424663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:31.425758 (+ 1095us) server_negotiation.cc:407] Connection header received
0620 21:50:31.427505 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:31.427508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:31.427573 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:31.427651 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:31.428979 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:31.429564 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:31.430755 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:31.430979 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:31.431095 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:31.433678 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:31.433681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:31.433684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:31.433709 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:31.435354 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:31.435892 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:31.435897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:31.435899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:31.435990 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:31.436344 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:31.436348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:31.436350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:31.436552 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:31.436652 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:31.437181 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 21:50:31.437319 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 21:50:31.438385 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:31.424960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44059 (local address 127.0.193.195:52289)
0620 21:50:31.425593 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:31.425649 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:31.427299 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:31.427832 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:31.427840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:31.428316 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:31.428827 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:31.428841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:31.429684 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:31.429687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:31.430627 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:31.430636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:31.431140 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:31.431144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:31.431368 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:31.431998 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:31.432014 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:31.433486 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:31.435475 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:31.435488 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:31.435504 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:31.435783 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:31.436087 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:31.436091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:31.436093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:31.436237 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:31.436663 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:31.436671 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:31.436932 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 21:50:31.437128 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 21:50:31.437476 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":160,"threads_started":1}
I20260620 21:50:31.439759 4306 heartbeater.cc:347] Connected to a master server at 127.0.193.254:44059
I20260620 21:50:31.440006 4306 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:31.440598 4306 heartbeater.cc:511] Master 127.0.193.254:44059 requested a full tablet report, sending...
I20260620 21:50:31.441634 3847 ts_manager.cc:194] Registered new tserver with Master: 7fbd094319ef44d2b8e121e90602bfdd (127.0.193.195:35181)
I20260620 21:50:31.442222 3847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:52289
I20260620 21:50:31.452939 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:50:31 dist-test-slave-kk5c krb5kdc[3802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992231, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 21:50:31 dist-test-slave-kk5c krb5kdc[3802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992231, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:31.481455 3886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:31.472215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.0.193.254:44059)
0620 21:50:31.472410 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:31.472415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:31.472561 (+ 146us) server_negotiation.cc:407] Connection header received
0620 21:50:31.472758 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:31.472761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:31.472830 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:31.472977 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:31.473894 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:31.474432 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:31.475359 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:31.475588 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:31.475704 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:31.478174 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:31.478180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:31.478187 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:31.478212 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:31.479650 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:31.480098 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:31.480104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:31.480105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:31.480214 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:31.480522 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:31.480526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:31.480527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:31.480764 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:31.480911 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:31.481152 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 21:50:31.481276 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 21:50:31.481902 3847 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34290
I20260620 21:50:31.482885 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 3909
I20260620 21:50:31.488577 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 4043
I20260620 21:50:31.494704 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 4177
I20260620 21:50:31.500447 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 3815
2026-06-20T21:50:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2366 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:50:31 dist-test-slave-kk5c krb5kdc[4318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:50:31 dist-test-slave-kk5c krb5kdc[4318](info): set up 2 sockets
Jun 20 21:50:31 dist-test-slave-kk5c krb5kdc[4318](info): commencing operation
krb5kdc: starting...
W20260620 21:50:32.563417 775 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:50:32 dist-test-slave-kk5c krb5kdc[4318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:50:32.698371 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:46429
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:46429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:32.819445 4331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:32.819813 4331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:32.819904 4331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:32.824311 4331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:32.824404 4331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:32.824440 4331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:32.824496 4331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:32.824533 4331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:32.830578 4331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:46429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:46429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:32.831900 4331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:32.832998 4331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:32.842404 4336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:32.842710 4337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:32.842725 4331 server_base.cc:1034] running on GCE node
W20260620 21:50:32.842796 4339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:32.843658 4331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:32.845033 4331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:32.846285 4331 hybrid_clock.cc:648] HybridClock initialized: now 1781992232846241 us; error 65 us; skew 500 ppm
Jun 20 21:50:32 dist-test-slave-kk5c krb5kdc[4318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:32.849928 4331 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:50:32.851114 4331 webserver.cc:460] Webserver started at http://127.0.193.254:39831/ using document root <none> and password file <none>
I20260620 21:50:32.851917 4331 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:32.852005 4331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:32.852262 4331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:32.854673 4331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "6043fdd0d2584b96b7f9da093dbf2312"
format_stamp: "Formatted at 2026-06-20 21:50:32 on dist-test-slave-kk5c"
server_key: "7cb0a6bad1d54726f857cda960a008ec"
server_key_iv: "4a3ba28844532dc8c0451cac30bd76b9"
server_key_version: "encryptionkey@0"
I20260620 21:50:32.855427 4331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "6043fdd0d2584b96b7f9da093dbf2312"
format_stamp: "Formatted at 2026-06-20 21:50:32 on dist-test-slave-kk5c"
server_key: "7cb0a6bad1d54726f857cda960a008ec"
server_key_iv: "4a3ba28844532dc8c0451cac30bd76b9"
server_key_version: "encryptionkey@0"
I20260620 21:50:32.859906 4331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:50:32.862869 4346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:32.864456 4331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:32.864593 4331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "6043fdd0d2584b96b7f9da093dbf2312"
format_stamp: "Formatted at 2026-06-20 21:50:32 on dist-test-slave-kk5c"
server_key: "7cb0a6bad1d54726f857cda960a008ec"
server_key_iv: "4a3ba28844532dc8c0451cac30bd76b9"
server_key_version: "encryptionkey@0"
I20260620 21:50:32.864759 4331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:32.881781 4331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:32.886016 4331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:32.886200 4331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:32.894860 4331 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:46429
I20260620 21:50:32.894872 4398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:46429 every 8 connection(s)
I20260620 21:50:32.896178 4331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:32.899480 4399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:32.905933 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 4331
I20260620 21:50:32.906049 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:32.905974 4399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312: Bootstrap starting.
I20260620 21:50:32.906340 775 external_mini_cluster.cc:1428] Setting key 569a8c90fbff6d0cd27de7834a8a22c6
I20260620 21:50:32.908903 4399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:32.909751 4399 log.cc:824] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:32.911892 4399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312: No bootstrap required, opened a new log
Jun 20 21:50:32 dist-test-slave-kk5c krb5kdc[4318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:32.916689 4399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6043fdd0d2584b96b7f9da093dbf2312" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46429 } }
I20260620 21:50:32.916981 4399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:32.917110 4399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6043fdd0d2584b96b7f9da093dbf2312, State: Initialized, Role: FOLLOWER
I20260620 21:50:32.917665 4399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6043fdd0d2584b96b7f9da093dbf2312" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46429 } }
I20260620 21:50:32.917827 4399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:32.917917 4399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:32.918038 4399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:32.918992 4399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6043fdd0d2584b96b7f9da093dbf2312" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46429 } }
I20260620 21:50:32.919409 4399 leader_election.cc:302] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6043fdd0d2584b96b7f9da093dbf2312; no voters:
I20260620 21:50:32.919823 4399 leader_election.cc:288] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:32.919898 4404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:32.920225 4404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [term 1 LEADER]: Becoming Leader. State: Replica: 6043fdd0d2584b96b7f9da093dbf2312, State: Running, Role: LEADER
I20260620 21:50:32.920614 4404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6043fdd0d2584b96b7f9da093dbf2312" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46429 } }
I20260620 21:50:32.921586 4399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:32.922516 4406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6043fdd0d2584b96b7f9da093dbf2312. Latest consensus state: current_term: 1 leader_uuid: "6043fdd0d2584b96b7f9da093dbf2312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6043fdd0d2584b96b7f9da093dbf2312" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46429 } } }
I20260620 21:50:32.922668 4406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:32.922765 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:32.907833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.0.193.254:46429)
0620 21:50:32.908294 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:32.908310 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:32.908347 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:50:32.909090 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:32.909108 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:32.909484 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:32.909777 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:32.910765 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:32.911928 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:32.913157 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:32.913647 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:32.913770 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:32.916889 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:32.916894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:32.916908 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:32.916940 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:32.918889 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:32.919445 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:32.919452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:32.919477 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:32.919651 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:32.919985 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:32.919988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:32.919991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:32.920535 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:32.921273 (+ 738us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:32.921444 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 21:50:32.921753 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":149,"threads_started":1}
I20260620 21:50:32.923076 4405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6043fdd0d2584b96b7f9da093dbf2312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6043fdd0d2584b96b7f9da093dbf2312" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46429 } } }
I20260620 21:50:32.923179 4405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:32.923460 4413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:32.926834 4413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:32.933429 4413 catalog_manager.cc:1347] Generated new cluster ID: 99d5794f4b2447509dfb74c04ba0caac
I20260620 21:50:32.933554 4413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:32.956034 4413 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:32.957316 4413 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:32.969543 4413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6043fdd0d2584b96b7f9da093dbf2312: Generated new TSK 0
I20260620 21:50:32.970516 4413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:50:33.002065 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46429
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:33.119109 4425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:33.119423 4425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:33.119469 4425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:33.124286 4425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:33.124372 4425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:33.124491 4425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:33.130312 4425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:33.131526 4425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:33.132593 4425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:33.143263 4431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:33.143273 4433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:33.143282 4430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:33.143327 4425 server_base.cc:1034] running on GCE node
I20260620 21:50:33.143929 4425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:33.144570 4425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:33.145745 4425 hybrid_clock.cc:648] HybridClock initialized: now 1781992233145727 us; error 36 us; skew 500 ppm
Jun 20 21:50:33 dist-test-slave-kk5c krb5kdc[4318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:33.148761 4425 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:50:33.149797 4425 webserver.cc:460] Webserver started at http://127.0.193.193:46615/ using document root <none> and password file <none>
I20260620 21:50:33.150393 4425 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:33.150456 4425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:33.150693 4425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:33.152856 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "6b960f65fceb4ba18ff61b45c0fde590"
format_stamp: "Formatted at 2026-06-20 21:50:33 on dist-test-slave-kk5c"
server_key: "929e27e808ce4c715930fd8f890d5488"
server_key_iv: "4ada82c5f362ddeea0797132794c7335"
server_key_version: "encryptionkey@0"
I20260620 21:50:33.153450 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "6b960f65fceb4ba18ff61b45c0fde590"
format_stamp: "Formatted at 2026-06-20 21:50:33 on dist-test-slave-kk5c"
server_key: "929e27e808ce4c715930fd8f890d5488"
server_key_iv: "4ada82c5f362ddeea0797132794c7335"
server_key_version: "encryptionkey@0"
I20260620 21:50:33.157298 4425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:33.159924 4440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:33.161384 4425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:33.161535 4425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "6b960f65fceb4ba18ff61b45c0fde590"
format_stamp: "Formatted at 2026-06-20 21:50:33 on dist-test-slave-kk5c"
server_key: "929e27e808ce4c715930fd8f890d5488"
server_key_iv: "4ada82c5f362ddeea0797132794c7335"
server_key_version: "encryptionkey@0"
I20260620 21:50:33.161655 4425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:33.187362 4425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:33.190725 4425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:33.190953 4425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:33.191623 4425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:33.192478 4425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:33.192525 4425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:33.192603 4425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:33.192636 4425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:33.201864 4425 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:45531
I20260620 21:50:33.202482 4553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:45531 every 8 connection(s)
I20260620 21:50:33.203050 4425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:33.209690 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 4425
I20260620 21:50:33.209818 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:50:33.210135 775 external_mini_cluster.cc:1428] Setting key b8b40dc222e4665b731ad7a5a3277ea2
Jun 20 21:50:33 dist-test-slave-kk5c krb5kdc[4318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:33.217710 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:33.204970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:58731 (local address 127.0.193.254:46429)
0620 21:50:33.205104 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:33.205107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:33.205910 (+ 803us) server_negotiation.cc:407] Connection header received
0620 21:50:33.207347 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:33.207350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:33.207397 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:33.207471 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:33.208657 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.209210 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.210772 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.211090 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.211216 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:33.213911 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:33.213914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:33.213916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:33.213941 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:33.215602 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:33.216151 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:33.216154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:33.216156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:33.216242 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:33.216595 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:33.216600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:33.216602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:33.216795 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:33.216916 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:33.217413 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 21:50:33.217542 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:50:33.218550 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:33.205339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:46429 (local address 127.0.193.193:58731)
0620 21:50:33.205760 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:33.205800 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:33.207062 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:33.207644 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:33.207650 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:33.208000 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:33.208508 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:33.208522 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.209314 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.209316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:33.210641 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:33.210650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.211288 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.211292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:33.211513 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:33.212150 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:33.212166 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:33.213731 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:33.215728 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:33.215734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:33.215752 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:33.216039 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:33.216338 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:33.216341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:33.216343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:33.216491 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:33.216899 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:33.216906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:33.217176 (+ 270us) client_negotiation.cc:769] Sending connection context
0620 21:50:33.217376 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 21:50:33.217722 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
I20260620 21:50:33.219909 4554 heartbeater.cc:347] Connected to a master server at 127.0.193.254:46429
I20260620 21:50:33.220166 4554 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:33.220712 4554 heartbeater.cc:511] Master 127.0.193.254:46429 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
I20260620 21:50:33.222515 4363 ts_manager.cc:194] Registered new tserver with Master: 6b960f65fceb4ba18ff61b45c0fde590 (127.0.193.193:45531)
I20260620 21:50:33.224102 4363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:58731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:50:33.239039 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46429
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:33.354997 4559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:33.355366 4559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:33.355461 4559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:33.360078 4559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:33.360195 4559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:33.360299 4559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:33.366214 4559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:33.367410 4559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:33.368516 4559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:33.381203 4565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:33.381301 4567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:33.381960 4564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:33.382022 4559 server_base.cc:1034] running on GCE node
I20260620 21:50:33.382464 4559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:33.383277 4559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:33.384502 4559 hybrid_clock.cc:648] HybridClock initialized: now 1781992233384478 us; error 38 us; skew 500 ppm
Jun 20 21:50:33 dist-test-slave-kk5c krb5kdc[4318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:33.387854 4559 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:50:33.388823 4559 webserver.cc:460] Webserver started at http://127.0.193.194:36351/ using document root <none> and password file <none>
I20260620 21:50:33.389415 4559 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:33.389478 4559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:33.389662 4559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:33.392066 4559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "9b0ddcaf24894e7a955c8500e5db1dc7"
format_stamp: "Formatted at 2026-06-20 21:50:33 on dist-test-slave-kk5c"
server_key: "f1e524f6963bb5075e4a750bea298f98"
server_key_iv: "7cbb83d89fb3e783e5f23a3308978ea1"
server_key_version: "encryptionkey@0"
I20260620 21:50:33.392686 4559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "9b0ddcaf24894e7a955c8500e5db1dc7"
format_stamp: "Formatted at 2026-06-20 21:50:33 on dist-test-slave-kk5c"
server_key: "f1e524f6963bb5075e4a750bea298f98"
server_key_iv: "7cbb83d89fb3e783e5f23a3308978ea1"
server_key_version: "encryptionkey@0"
I20260620 21:50:33.397420 4559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:33.400317 4574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:33.402047 4559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:50:33.402170 4559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "9b0ddcaf24894e7a955c8500e5db1dc7"
format_stamp: "Formatted at 2026-06-20 21:50:33 on dist-test-slave-kk5c"
server_key: "f1e524f6963bb5075e4a750bea298f98"
server_key_iv: "7cbb83d89fb3e783e5f23a3308978ea1"
server_key_version: "encryptionkey@0"
I20260620 21:50:33.402272 4559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:33.422353 4559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:33.426185 4559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:33.426357 4559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:33.426906 4559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:33.427875 4559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:33.427924 4559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:33.427974 4559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:33.427991 4559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:33.437736 4559 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:41939
I20260620 21:50:33.437755 4687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:41939 every 8 connection(s)
I20260620 21:50:33.438694 4559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:33.446527 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 4559
I20260620 21:50:33.446682 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:50:33.446976 775 external_mini_cluster.cc:1428] Setting key dbcf0edcbc119f2d74605f21c003a5b2
Jun 20 21:50:33 dist-test-slave-kk5c krb5kdc[4318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:33.454311 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:33.440794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:58553 (local address 127.0.193.254:46429)
0620 21:50:33.440970 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:33.440974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:33.441880 (+ 906us) server_negotiation.cc:407] Connection header received
0620 21:50:33.443361 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:33.443368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:33.443420 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:33.443500 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:33.444756 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.445284 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.446526 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.446855 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.446966 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:33.450115 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:33.450123 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:33.450126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:33.450153 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:33.451868 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:33.452555 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:33.452560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:33.452562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:33.452652 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:33.453130 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:33.453135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:33.453138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:33.453352 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:33.453471 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:33.454010 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 21:50:33.454147 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 21:50:33.455296 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:33.441189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:46429 (local address 127.0.193.194:58553)
0620 21:50:33.441715 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:33.441762 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:33.443052 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:33.443709 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:33.443716 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:33.444088 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:33.444579 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:33.444593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.445397 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.445399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:33.446376 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:33.446385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.447022 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.447026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:33.447271 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:33.448036 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:33.448056 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:33.449781 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:33.452053 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:33.452060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:33.452079 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:33.452395 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:33.452753 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:33.452756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:33.452758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:33.452998 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:33.453469 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:33.453478 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:33.453755 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 21:50:33.453958 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 21:50:33.454307 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":141,"threads_started":1}
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
I20260620 21:50:33.456758 4688 heartbeater.cc:347] Connected to a master server at 127.0.193.254:46429
I20260620 21:50:33.457022 4688 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:33.457602 4688 heartbeater.cc:511] Master 127.0.193.254:46429 requested a full tablet report, sending...
I20260620 21:50:33.458652 4363 ts_manager.cc:194] Registered new tserver with Master: 9b0ddcaf24894e7a955c8500e5db1dc7 (127.0.193.194:41939)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
I20260620 21:50:33.459479 4363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:58553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:50:33.475610 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46429
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:33.590869 4693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:33.591146 4693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:33.591189 4693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:33.595398 4693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:33.595470 4693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:33.595573 4693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:33.601738 4693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:33.602962 4693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:33.604143 4693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:33.617990 4701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:33.618055 4698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:33.618055 4693 server_base.cc:1034] running on GCE node
W20260620 21:50:33.618271 4699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:33.618722 4693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:33.619616 4693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:33.620818 4693 hybrid_clock.cc:648] HybridClock initialized: now 1781992233620791 us; error 41 us; skew 500 ppm
Jun 20 21:50:33 dist-test-slave-kk5c krb5kdc[4318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:33.624524 4693 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:50:33.625749 4693 webserver.cc:460] Webserver started at http://127.0.193.195:37295/ using document root <none> and password file <none>
I20260620 21:50:33.626353 4693 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:33.626420 4693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:33.626647 4693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:33.628940 4693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "0d1f7bfeab124c35adf351f2110663de"
format_stamp: "Formatted at 2026-06-20 21:50:33 on dist-test-slave-kk5c"
server_key: "d79060086908eba0205520877a943e35"
server_key_iv: "99deb1f9723f2081e354d9790e9dce09"
server_key_version: "encryptionkey@0"
I20260620 21:50:33.629585 4693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "0d1f7bfeab124c35adf351f2110663de"
format_stamp: "Formatted at 2026-06-20 21:50:33 on dist-test-slave-kk5c"
server_key: "d79060086908eba0205520877a943e35"
server_key_iv: "99deb1f9723f2081e354d9790e9dce09"
server_key_version: "encryptionkey@0"
I20260620 21:50:33.634397 4693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:50:33.637322 4708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:33.638967 4693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 21:50:33.639150 4693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "0d1f7bfeab124c35adf351f2110663de"
format_stamp: "Formatted at 2026-06-20 21:50:33 on dist-test-slave-kk5c"
server_key: "d79060086908eba0205520877a943e35"
server_key_iv: "99deb1f9723f2081e354d9790e9dce09"
server_key_version: "encryptionkey@0"
I20260620 21:50:33.639374 4693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:33.655949 4693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:33.659665 4693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:33.659889 4693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:33.660506 4693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:33.661441 4693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:33.661515 4693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:33.661586 4693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:33.661645 4693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:33.671700 4693 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:38815
I20260620 21:50:33.671743 4821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:38815 every 8 connection(s)
I20260620 21:50:33.672664 4693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
Jun 20 21:50:33 dist-test-slave-kk5c krb5kdc[4318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:33.682960 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 4693
I20260620 21:50:33.683096 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:50:33.683472 775 external_mini_cluster.cc:1428] Setting key fdba4a224322c18a0a7f0aad50be141f
I20260620 21:50:33.687439 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:33.674527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:36365 (local address 127.0.193.254:46429)
0620 21:50:33.674678 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:33.674682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:33.675564 (+ 882us) server_negotiation.cc:407] Connection header received
0620 21:50:33.676916 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:33.676919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:33.676977 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:33.677084 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:33.678341 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.678876 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.680126 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.680347 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.680459 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:33.683250 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:33.683256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:33.683258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:33.683291 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:33.685076 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:33.685621 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:33.685626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:33.685628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:33.685739 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:33.686111 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:33.686114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:33.686115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:33.686315 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:33.686395 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:33.687026 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 21:50:33.687155 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:50:33.688196 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:33.674888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:46429 (local address 127.0.193.195:36365)
0620 21:50:33.675423 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:33.675465 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:33.676703 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:33.677252 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:33.677259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:33.677623 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:33.678181 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:33.678197 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.679005 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.679009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:33.679983 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:33.679993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.680500 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.680504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:33.680680 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:33.681384 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:33.681443 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:33.683002 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:33.685183 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:33.685188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:33.685202 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:33.685475 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:33.685842 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:33.685844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:33.685846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:33.686009 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:33.686481 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:33.686494 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:33.686748 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 21:50:33.686933 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 21:50:33.687285 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":126,"threads_started":1}
I20260620 21:50:33.689394 4822 heartbeater.cc:347] Connected to a master server at 127.0.193.254:46429
I20260620 21:50:33.689743 4822 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:33.690299 4822 heartbeater.cc:511] Master 127.0.193.254:46429 requested a full tablet report, sending...
I20260620 21:50:33.691581 4363 ts_manager.cc:194] Registered new tserver with Master: 0d1f7bfeab124c35adf351f2110663de (127.0.193.195:38815)
I20260620 21:50:33.692143 4363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:36365
I20260620 21:50:33.697844 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:50:33 dist-test-slave-kk5c krb5kdc[4318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:50:33 dist-test-slave-kk5c krb5kdc[4318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.193@KRBTEST.COM
I20260620 21:50:33.742254 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:33.731230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53650 (local address 127.0.193.193:45531)
0620 21:50:33.731628 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:33.731639 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:33.731669 (+ 30us) server_negotiation.cc:407] Connection header received
0620 21:50:33.731732 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:33.731742 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:33.732001 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:33.732164 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:33.733002 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.733803 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.734800 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.735279 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.735402 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:33.738148 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:33.738158 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:33.738173 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:33.738200 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:33.740040 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:33.740586 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:33.740592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:33.740593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:33.740703 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:33.741031 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:33.741035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:33.741036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:33.741407 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:33.741593 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:33.741824 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 21:50:33.742025 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":148,"threads_started":1}
Jun 20 21:50:33 dist-test-slave-kk5c krb5kdc[4318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:33.755108 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:33.745780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59166 (local address 127.0.193.254:46429)
0620 21:50:33.745917 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:33.745920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:33.745934 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:50:33.746031 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:33.746033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:33.746074 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:33.746140 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:33.746859 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.747513 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.748504 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:33.748742 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:33.748839 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:33.751868 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:33.751872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:33.751874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:33.751906 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:33.753317 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:33.753787 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:33.753791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:33.753792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:33.753867 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:33.754148 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:33.754152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:33.754153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:33.754395 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:33.754535 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:33.754802 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 21:50:33.754926 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 21:50:33.755568 4363 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59166
I20260620 21:50:33.756944 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 4425
I20260620 21:50:33.762758 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 4559
I20260620 21:50:33.769035 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 4693
I20260620 21:50:33.774747 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 4331
2026-06-20T21:50:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2274 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:50:35 dist-test-slave-kk5c krb5kdc[4834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:50:35 dist-test-slave-kk5c krb5kdc[4834](info): set up 2 sockets
Jun 20 21:50:35 dist-test-slave-kk5c krb5kdc[4834](info): commencing operation
krb5kdc: starting...
W20260620 21:50:36.583534 775 mini_kdc.cc:121] Time spent starting KDC: real 2.786s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:50:36 dist-test-slave-kk5c krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:50:36.738924 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:42823
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:42823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:36.856829 4847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:36.857134 4847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:36.857179 4847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:36.861560 4847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:36.861656 4847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:36.861677 4847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:36.861696 4847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:36.861711 4847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:36.869509 4847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:42823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:42823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:36.870914 4847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:36.872279 4847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:36.881928 4855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:36.881947 4852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:36.881945 4853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:36.882308 4847 server_base.cc:1034] running on GCE node
I20260620 21:50:36.883069 4847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:36.884337 4847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:36.885556 4847 hybrid_clock.cc:648] HybridClock initialized: now 1781992236885539 us; error 31 us; skew 500 ppm
Jun 20 21:50:36 dist-test-slave-kk5c krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:36.888924 4847 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:50:36.889950 4847 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:36.890055 4847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:36.890383 4847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:36.892653 4847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "e6f0d56d2a4749acb9af49b43269b05d"
format_stamp: "Formatted at 2026-06-20 21:50:36 on dist-test-slave-kk5c"
server_key: "35a6a3f44df07e66c706e4abc9d5fdc7"
server_key_iv: "9aa2c5b1034d774602096365a7b22554"
server_key_version: "encryptionkey@0"
I20260620 21:50:36.893263 4847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "e6f0d56d2a4749acb9af49b43269b05d"
format_stamp: "Formatted at 2026-06-20 21:50:36 on dist-test-slave-kk5c"
server_key: "35a6a3f44df07e66c706e4abc9d5fdc7"
server_key_iv: "9aa2c5b1034d774602096365a7b22554"
server_key_version: "encryptionkey@0"
I20260620 21:50:36.897392 4847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:36.900079 4861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:36.901463 4847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:36.901600 4847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "e6f0d56d2a4749acb9af49b43269b05d"
format_stamp: "Formatted at 2026-06-20 21:50:36 on dist-test-slave-kk5c"
server_key: "35a6a3f44df07e66c706e4abc9d5fdc7"
server_key_iv: "9aa2c5b1034d774602096365a7b22554"
server_key_version: "encryptionkey@0"
I20260620 21:50:36.901714 4847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:36.958729 4847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:36.962451 4847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:36.962677 4847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:36.970291 4847 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:42823
I20260620 21:50:36.970587 4913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:42823 every 8 connection(s)
I20260620 21:50:36.971046 4847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:36.974464 4914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:36.977249 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 4847
I20260620 21:50:36.977375 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:36.977703 775 external_mini_cluster.cc:1428] Setting key 1f8c89de67da544ced2cce81e3ffd7ed
I20260620 21:50:36.981340 4914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d: Bootstrap starting.
I20260620 21:50:36.984148 4914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:36.984925 4914 log.cc:824] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:36.987687 4914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d: No bootstrap required, opened a new log
Jun 20 21:50:36 dist-test-slave-kk5c krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:36.992004 4914 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f0d56d2a4749acb9af49b43269b05d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42823 } }
I20260620 21:50:36.992283 4914 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:36.992323 4914 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f0d56d2a4749acb9af49b43269b05d, State: Initialized, Role: FOLLOWER
I20260620 21:50:36.992952 4914 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f0d56d2a4749acb9af49b43269b05d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42823 } }
I20260620 21:50:36.993177 4914 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:36.993274 4914 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:36.993429 4914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:36.994692 4914 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f0d56d2a4749acb9af49b43269b05d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42823 } }
I20260620 21:50:36.995306 4914 leader_election.cc:302] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6f0d56d2a4749acb9af49b43269b05d; no voters:
I20260620 21:50:36.995874 4914 leader_election.cc:288] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:36.995970 4919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:36.996173 4919 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [term 1 LEADER]: Becoming Leader. State: Replica: e6f0d56d2a4749acb9af49b43269b05d, State: Running, Role: LEADER
I20260620 21:50:36.996366 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:36.979412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.0.193.254:42823)
0620 21:50:36.979914 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:36.979933 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:36.980030 (+ 97us) server_negotiation.cc:407] Connection header received
0620 21:50:36.980814 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:36.980836 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:36.981260 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:36.981642 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:36.982407 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:36.983929 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:36.985119 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:36.985891 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:36.986092 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:36.989456 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:36.989461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:36.989477 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:36.989508 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:36.992484 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:36.993121 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:36.993128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:36.993136 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:36.993340 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:36.993735 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:36.993737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:36.993739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:36.994193 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:36.994357 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:36.994763 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 21:50:36.995111 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":131,"threads_started":1}
I20260620 21:50:36.996618 4919 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f0d56d2a4749acb9af49b43269b05d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42823 } }
I20260620 21:50:36.997426 4914 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:36.997925 4919 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6f0d56d2a4749acb9af49b43269b05d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f0d56d2a4749acb9af49b43269b05d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42823 } } }
I20260620 21:50:36.998055 4919 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:36.998735 4923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:37.000504 4920 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6f0d56d2a4749acb9af49b43269b05d. Latest consensus state: current_term: 1 leader_uuid: "e6f0d56d2a4749acb9af49b43269b05d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6f0d56d2a4749acb9af49b43269b05d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42823 } } }
I20260620 21:50:37.000626 4920 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:37.002013 4923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:37.008641 4923 catalog_manager.cc:1347] Generated new cluster ID: 7fdad9466e844bdaa9e643e66cd375d2
I20260620 21:50:37.008726 4923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:37.033390 4923 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:37.034418 4923 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:37.042313 4923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d: Generated new TSK 0
I20260620 21:50:37.043205 4923 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:50:37.089300 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:42823
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:37.207398 4940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:37.207707 4940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:37.207783 4940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:37.212080 4940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:37.212181 4940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:37.212280 4940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:37.217815 4940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:42823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:37.219008 4940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:37.220201 4940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:37.230768 4946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:37.230823 4945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:37.230760 4948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:37.231127 4940 server_base.cc:1034] running on GCE node
I20260620 21:50:37.231601 4940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:37.232378 4940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:37.233537 4940 hybrid_clock.cc:648] HybridClock initialized: now 1781992237233514 us; error 36 us; skew 500 ppm
Jun 20 21:50:37 dist-test-slave-kk5c krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:37.237136 4940 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:50:37.237950 4940 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:37.238060 4940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:37.238300 4940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:37.240638 4940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "56ffe45fb8e9443f8b8d5cba801fcbee"
format_stamp: "Formatted at 2026-06-20 21:50:37 on dist-test-slave-kk5c"
server_key: "de288f36bfe0634d497259f91a76611a"
server_key_iv: "1c96fdeae6448a1fa82879ce153f7f2a"
server_key_version: "encryptionkey@0"
I20260620 21:50:37.241240 4940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "56ffe45fb8e9443f8b8d5cba801fcbee"
format_stamp: "Formatted at 2026-06-20 21:50:37 on dist-test-slave-kk5c"
server_key: "de288f36bfe0634d497259f91a76611a"
server_key_iv: "1c96fdeae6448a1fa82879ce153f7f2a"
server_key_version: "encryptionkey@0"
I20260620 21:50:37.245301 4940 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:37.248080 4954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:37.249579 4940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:37.249735 4940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "56ffe45fb8e9443f8b8d5cba801fcbee"
format_stamp: "Formatted at 2026-06-20 21:50:37 on dist-test-slave-kk5c"
server_key: "de288f36bfe0634d497259f91a76611a"
server_key_iv: "1c96fdeae6448a1fa82879ce153f7f2a"
server_key_version: "encryptionkey@0"
I20260620 21:50:37.249863 4940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:37.262856 4940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:37.267395 4940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:37.267629 4940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:37.268253 4940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:37.269217 4940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:37.269291 4940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:37.269366 4940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:37.269408 4940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:37.279592 4940 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:32813
I20260620 21:50:37.279624 5067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:32813 every 8 connection(s)
I20260620 21:50:37.280263 4940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:37.287258 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 4940
I20260620 21:50:37.287386 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:50:37.287671 775 external_mini_cluster.cc:1428] Setting key f402a51c95ca4967635873d3305c4b30
Jun 20 21:50:37 dist-test-slave-kk5c krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:37.296505 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.282253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:35473 (local address 127.0.193.254:42823)
0620 21:50:37.282441 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:37.282445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:37.283597 (+ 1152us) server_negotiation.cc:407] Connection header received
0620 21:50:37.285175 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:37.285179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:37.285246 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:37.285363 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:37.286739 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.287390 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.289225 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.289468 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.289594 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.292433 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:37.292437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:37.292439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:37.292466 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:37.294028 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:37.294629 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:37.294637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:37.294639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:37.294719 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:37.295259 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:37.295263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:37.295265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:37.295494 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:37.295617 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:37.296180 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 21:50:37.296329 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 21:50:37.297521 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.282714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:42823 (local address 127.0.193.193:35473)
0620 21:50:37.283424 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:37.283471 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:37.284751 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:37.285571 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:37.285579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:37.286031 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:37.286559 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.286577 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.287540 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.287543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.288940 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.288971 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.289642 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.289647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.289838 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.290538 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:37.290555 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:37.292217 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:37.294178 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:37.294184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:37.294204 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:37.294510 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:37.294822 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:37.294825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:37.294827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:37.295085 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:37.295617 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:37.295626 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:37.295921 (+ 295us) client_negotiation.cc:769] Sending connection context
0620 21:50:37.296124 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 21:50:37.296499 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":306,"threads_started":1}
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
I20260620 21:50:37.299108 5068 heartbeater.cc:347] Connected to a master server at 127.0.193.254:42823
I20260620 21:50:37.299574 5068 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:37.300045 5068 heartbeater.cc:511] Master 127.0.193.254:42823 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
I20260620 21:50:37.301791 4876 ts_manager.cc:194] Registered new tserver with Master: 56ffe45fb8e9443f8b8d5cba801fcbee (127.0.193.193:32813)
I20260620 21:50:37.303400 4876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:35473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:50:37.320003 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:42823
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:37.435122 5073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:37.435529 5073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:37.435616 5073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:37.440181 5073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:37.440266 5073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:37.440335 5073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:37.446201 5073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:42823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:37.447360 5073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:37.448434 5073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:37.458964 5079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:37.458969 5078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:37.458964 5081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:37.459434 5073 server_base.cc:1034] running on GCE node
I20260620 21:50:37.459969 5073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:37.460768 5073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:37.461952 5073 hybrid_clock.cc:648] HybridClock initialized: now 1781992237461928 us; error 30 us; skew 500 ppm
Jun 20 21:50:37 dist-test-slave-kk5c krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:37.465032 5073 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:50:37.465894 5073 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:37.465971 5073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:37.466202 5073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:37.468523 5073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "b601e8928a894208922d72e3fb5dd8ee"
format_stamp: "Formatted at 2026-06-20 21:50:37 on dist-test-slave-kk5c"
server_key: "79d9ee6daa30b743f57e95b3aa6f8549"
server_key_iv: "7fff1b6bf81af8d6deb2eaf6eb337682"
server_key_version: "encryptionkey@0"
I20260620 21:50:37.469269 5073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "b601e8928a894208922d72e3fb5dd8ee"
format_stamp: "Formatted at 2026-06-20 21:50:37 on dist-test-slave-kk5c"
server_key: "79d9ee6daa30b743f57e95b3aa6f8549"
server_key_iv: "7fff1b6bf81af8d6deb2eaf6eb337682"
server_key_version: "encryptionkey@0"
I20260620 21:50:37.473470 5073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:37.476177 5087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:37.477571 5073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:37.477721 5073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "b601e8928a894208922d72e3fb5dd8ee"
format_stamp: "Formatted at 2026-06-20 21:50:37 on dist-test-slave-kk5c"
server_key: "79d9ee6daa30b743f57e95b3aa6f8549"
server_key_iv: "7fff1b6bf81af8d6deb2eaf6eb337682"
server_key_version: "encryptionkey@0"
I20260620 21:50:37.477849 5073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:37.537905 5073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:37.541427 5073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:37.541680 5073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:37.542338 5073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:37.543326 5073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:37.543398 5073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:37.543495 5073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:37.543533 5073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:37.554121 5073 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:36143
I20260620 21:50:37.554225 5200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:36143 every 8 connection(s)
I20260620 21:50:37.554880 5073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:37.557984 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 5073
I20260620 21:50:37.558133 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:50:37.558554 775 external_mini_cluster.cc:1428] Setting key 53f3c447801a9d69df54bf998045af63
Jun 20 21:50:37 dist-test-slave-kk5c krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
I20260620 21:50:37.572184 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.557377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:50227 (local address 127.0.193.254:42823)
0620 21:50:37.557528 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:37.557532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:37.558649 (+ 1117us) server_negotiation.cc:407] Connection header received
0620 21:50:37.560459 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:37.560462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:37.560503 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:37.560574 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:37.562438 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.563369 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.564716 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.565177 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.565314 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.568246 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:37.568250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:37.568252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:37.568282 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:37.569752 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:37.570351 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:37.570354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:37.570356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:37.570446 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:37.570922 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:37.570925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:37.570926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:37.571127 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:37.571314 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:37.571931 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 21:50:37.572053 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:50:37.573516 5203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.557822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:42823 (local address 127.0.193.194:50227)
0620 21:50:37.558427 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:37.558492 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:37.560206 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:37.560771 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:37.560784 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:37.561392 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:37.562163 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.562191 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.563534 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.563537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.564568 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.564578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.565346 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.565349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.565514 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.566227 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:37.566246 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:37.568045 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:37.569881 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:37.569888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:37.569903 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:37.570199 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:37.570573 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:37.570580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:37.570585 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:37.570798 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:37.571294 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:37.571304 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:37.571640 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 21:50:37.571854 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 21:50:37.572260 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":152,"threads_started":1}
I20260620 21:50:37.574853 5201 heartbeater.cc:347] Connected to a master server at 127.0.193.254:42823
I20260620 21:50:37.575129 5201 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:37.575636 5201 heartbeater.cc:511] Master 127.0.193.254:42823 requested a full tablet report, sending...
I20260620 21:50:37.576725 4876 ts_manager.cc:194] Registered new tserver with Master: b601e8928a894208922d72e3fb5dd8ee (127.0.193.194:36143)
I20260620 21:50:37.577334 4876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:50227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:50:37.592244 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:42823
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:37.722050 5206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:37.722369 5206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:37.722477 5206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:37.727042 5206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:37.727131 5206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:37.727262 5206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:37.733265 5206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:42823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:37.734442 5206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:37.735632 5206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:37.745537 5214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:37.745529 5212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:37.745586 5206 server_base.cc:1034] running on GCE node
W20260620 21:50:37.745529 5211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:37.746234 5206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:37.746882 5206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:37.748093 5206 hybrid_clock.cc:648] HybridClock initialized: now 1781992237748058 us; error 51 us; skew 500 ppm
Jun 20 21:50:37 dist-test-slave-kk5c krb5kdc[4834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:37.751381 5206 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:50:37.752214 5206 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:37.752322 5206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:37.752584 5206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:37.755122 5206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "522cda7dc47346ab890af10c3c1deeb2"
format_stamp: "Formatted at 2026-06-20 21:50:37 on dist-test-slave-kk5c"
server_key: "6b287dc6f8d41739dff9e35153222011"
server_key_iv: "f10164ad705a8ccc5c812810332b4894"
server_key_version: "encryptionkey@0"
I20260620 21:50:37.755810 5206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "522cda7dc47346ab890af10c3c1deeb2"
format_stamp: "Formatted at 2026-06-20 21:50:37 on dist-test-slave-kk5c"
server_key: "6b287dc6f8d41739dff9e35153222011"
server_key_iv: "f10164ad705a8ccc5c812810332b4894"
server_key_version: "encryptionkey@0"
I20260620 21:50:37.760018 5206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:50:37.762718 5220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:37.764465 5206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:37.764618 5206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "522cda7dc47346ab890af10c3c1deeb2"
format_stamp: "Formatted at 2026-06-20 21:50:37 on dist-test-slave-kk5c"
server_key: "6b287dc6f8d41739dff9e35153222011"
server_key_iv: "f10164ad705a8ccc5c812810332b4894"
server_key_version: "encryptionkey@0"
I20260620 21:50:37.764750 5206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:37.792994 5206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:37.796530 5206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:37.796747 5206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:37.797434 5206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:37.798448 5206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:37.798521 5206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:37.798592 5206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:37.798646 5206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:37.808082 5206 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:33867
I20260620 21:50:37.808099 5333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:33867 every 8 connection(s)
I20260620 21:50:37.808712 5206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:50:37.810423 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 5206
I20260620 21:50:37.810596 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:50:37.810922 775 external_mini_cluster.cc:1428] Setting key 410257ecd2fe3d13f5d3c97b79080a3b
Jun 20 21:50:37 dist-test-slave-kk5c krb5kdc[4834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:37.824153 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.810548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:49115 (local address 127.0.193.254:42823)
0620 21:50:37.810690 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:37.810695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:37.811732 (+ 1037us) server_negotiation.cc:407] Connection header received
0620 21:50:37.813550 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:37.813552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:37.813595 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:37.813662 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:37.814905 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.815496 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.816781 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.817028 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.817126 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.819922 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:37.819926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:37.819928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:37.819968 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:37.821680 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:37.822274 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:37.822276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:37.822278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:37.822398 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:37.822810 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:37.822812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:37.822813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:37.823008 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:37.823090 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:37.823823 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 21:50:37.823971 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:50:37.825177 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.810939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:42823 (local address 127.0.193.195:49115)
0620 21:50:37.811546 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:37.811607 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:37.813351 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:37.813835 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:37.813842 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:37.814252 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:37.814751 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.814764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.815616 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.815619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.816614 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.816624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.817182 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.817185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.817344 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.818048 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:37.818066 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:37.819727 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:37.821831 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:37.821837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:37.821851 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:37.822177 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:37.822518 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:37.822520 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:37.822528 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:37.822684 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:37.823117 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:37.823123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:37.823473 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 21:50:37.823705 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 21:50:37.824108 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":168,"threads_started":1}
I20260620 21:50:37.826350 5334 heartbeater.cc:347] Connected to a master server at 127.0.193.254:42823
I20260620 21:50:37.826623 5334 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:37.827118 5334 heartbeater.cc:511] Master 127.0.193.254:42823 requested a full tablet report, sending...
I20260620 21:50:37.828296 4876 ts_manager.cc:194] Registered new tserver with Master: 522cda7dc47346ab890af10c3c1deeb2 (127.0.193.195:33867)
I20260620 21:50:37.829035 4876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:49115
I20260620 21:50:37.837888 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:50:37.848269 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.840097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52888 (local address 127.0.193.254:42823)
0620 21:50:37.840270 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:37.840275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:37.840440 (+ 165us) server_negotiation.cc:407] Connection header received
0620 21:50:37.840571 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:37.840574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:37.840618 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:37.840689 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:37.841517 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.842066 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.842949 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.843164 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.843331 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.844719 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:37.844722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:37.844724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:37.844749 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:37.846329 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:37.846838 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:37.846841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:37.846842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:37.846917 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:37.847327 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:37.847330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:37.847331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:37.847565 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:37.847675 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:37.847930 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 21:50:37.848038 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:50:37.851496 4876 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:50:37.854750 4876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:37.870043 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.863301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:32813 (local address 127.0.0.1:48892)
0620 21:50:37.863692 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:37.863733 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:37.863849 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:37.865299 (+ 1450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:37.865302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:37.865320 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:37.865433 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.865440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.866817 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.866821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.868267 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.868278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.869397 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.869401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.869520 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.869582 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 21:50:37.869691 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 21:50:37.869850 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":75,"thread_start_us":108,"threads_started":1}
I20260620 21:50:37.870196 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.864113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:36143 (local address 127.0.0.1:36894)
0620 21:50:37.864328 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:37.864343 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:37.864426 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:37.865234 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:37.865240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:37.865263 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:37.865421 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.865428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.866599 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.866605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.868272 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.868278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.869699 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.869702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.869789 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.869812 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:50:37.869877 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 21:50:37.869994 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":138,"thread_start_us":100,"threads_started":1}
I20260620 21:50:37.870205 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.864248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.0.193.194:36143)
0620 21:50:37.864670 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:37.864683 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:37.864718 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:50:37.864807 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:37.864814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:37.865038 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:37.865236 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:37.865551 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.866480 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.868760 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.869566 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.869701 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.869903 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:37.869960 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 21:50:37.870054 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":107,"threads_started":1}
I20260620 21:50:37.870241 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.864463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:33867 (local address 127.0.0.1:50408)
0620 21:50:37.864693 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:37.864707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:37.864786 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:37.866082 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:37.866085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:37.866105 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:37.866203 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.866208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.867457 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.867460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.868537 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.868547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.869832 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.869835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.869949 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.869971 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:50:37.870038 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 21:50:37.870121 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":7,"thread_start_us":112,"threads_started":1}
I20260620 21:50:37.870656 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.864711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.0.193.195:33867)
0620 21:50:37.865654 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:37.865665 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:37.865685 (+ 20us) server_negotiation.cc:407] Connection header received
0620 21:50:37.865752 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:37.865757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:37.865930 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:37.866075 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:37.866325 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.867289 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.868660 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.869512 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.869617 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.869772 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:37.870429 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 21:50:37.870535 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":122,"threads_started":1}
I20260620 21:50:37.871299 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.863422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.0.193.193:32813)
0620 21:50:37.864872 (+ 1450us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:37.864879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:37.864900 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:50:37.864970 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:37.864973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:37.865155 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:37.865311 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:37.865566 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.866679 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.868398 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.869264 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.869379 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.869540 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:37.871028 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0620 21:50:37.871135 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1310,"thread_start_us":57,"threads_started":1}
I20260620 21:50:37.873734 5268 tablet_service.cc:1467] Processing CreateTablet for tablet 9c187ed548d5422a98006a251b622da6 (DEFAULT_TABLE table=test-table [id=089abb0a90874a7289a515aa7d46901b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:37.873783 5001 tablet_service.cc:1467] Processing CreateTablet for tablet 9c187ed548d5422a98006a251b622da6 (DEFAULT_TABLE table=test-table [id=089abb0a90874a7289a515aa7d46901b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:37.873790 5135 tablet_service.cc:1467] Processing CreateTablet for tablet 9c187ed548d5422a98006a251b622da6 (DEFAULT_TABLE table=test-table [id=089abb0a90874a7289a515aa7d46901b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:37.874873 5135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c187ed548d5422a98006a251b622da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:37.874874 5001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c187ed548d5422a98006a251b622da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:37.874888 5268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c187ed548d5422a98006a251b622da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:37.881536 5351 tablet_bootstrap.cc:492] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2: Bootstrap starting.
I20260620 21:50:37.882777 5352 tablet_bootstrap.cc:492] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee: Bootstrap starting.
I20260620 21:50:37.883569 5353 tablet_bootstrap.cc:492] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee: Bootstrap starting.
I20260620 21:50:37.884541 5351 tablet_bootstrap.cc:654] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:37.884757 5352 tablet_bootstrap.cc:654] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:37.885532 5352 log.cc:824] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:37.885730 5353 tablet_bootstrap.cc:654] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:37.885937 5351 log.cc:824] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:37.886972 5353 log.cc:824] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:37.888070 5351 tablet_bootstrap.cc:492] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2: No bootstrap required, opened a new log
I20260620 21:50:37.888069 5352 tablet_bootstrap.cc:492] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee: No bootstrap required, opened a new log
I20260620 21:50:37.888278 5352 ts_tablet_manager.cc:1397] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:50:37.888285 5351 ts_tablet_manager.cc:1397] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 21:50:37.888764 5353 tablet_bootstrap.cc:492] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee: No bootstrap required, opened a new log
I20260620 21:50:37.888965 5353 ts_tablet_manager.cc:1397] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:50:37.892122 5353 raft_consensus.cc:348] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ffe45fb8e9443f8b8d5cba801fcbee" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 32813 } } peers { permanent_uuid: "b601e8928a894208922d72e3fb5dd8ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36143 } } peers { permanent_uuid: "522cda7dc47346ab890af10c3c1deeb2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33867 } }
I20260620 21:50:37.892413 5353 raft_consensus.cc:374] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:37.892477 5353 raft_consensus.cc:729] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b601e8928a894208922d72e3fb5dd8ee, State: Initialized, Role: FOLLOWER
I20260620 21:50:37.892511 5351 raft_consensus.cc:348] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ffe45fb8e9443f8b8d5cba801fcbee" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 32813 } } peers { permanent_uuid: "b601e8928a894208922d72e3fb5dd8ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36143 } } peers { permanent_uuid: "522cda7dc47346ab890af10c3c1deeb2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33867 } }
I20260620 21:50:37.892810 5351 raft_consensus.cc:374] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:37.892863 5351 raft_consensus.cc:729] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522cda7dc47346ab890af10c3c1deeb2, State: Initialized, Role: FOLLOWER
I20260620 21:50:37.892863 5352 raft_consensus.cc:348] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ffe45fb8e9443f8b8d5cba801fcbee" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 32813 } } peers { permanent_uuid: "b601e8928a894208922d72e3fb5dd8ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36143 } } peers { permanent_uuid: "522cda7dc47346ab890af10c3c1deeb2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33867 } }
I20260620 21:50:37.893095 5352 raft_consensus.cc:374] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:37.893138 5352 raft_consensus.cc:729] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ffe45fb8e9443f8b8d5cba801fcbee, State: Initialized, Role: FOLLOWER
I20260620 21:50:37.893010 5353 consensus_queue.cc:260] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ffe45fb8e9443f8b8d5cba801fcbee" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 32813 } } peers { permanent_uuid: "b601e8928a894208922d72e3fb5dd8ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36143 } } peers { permanent_uuid: "522cda7dc47346ab890af10c3c1deeb2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33867 } }
I20260620 21:50:37.893323 5351 consensus_queue.cc:260] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ffe45fb8e9443f8b8d5cba801fcbee" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 32813 } } peers { permanent_uuid: "b601e8928a894208922d72e3fb5dd8ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36143 } } peers { permanent_uuid: "522cda7dc47346ab890af10c3c1deeb2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33867 } }
I20260620 21:50:37.893559 5352 consensus_queue.cc:260] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ffe45fb8e9443f8b8d5cba801fcbee" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 32813 } } peers { permanent_uuid: "b601e8928a894208922d72e3fb5dd8ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36143 } } peers { permanent_uuid: "522cda7dc47346ab890af10c3c1deeb2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33867 } }
I20260620 21:50:37.894173 5353 ts_tablet_manager.cc:1428] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 21:50:37.894259 5351 ts_tablet_manager.cc:1428] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:50:37.894444 5201 heartbeater.cc:503] Master 127.0.193.254:42823 was elected leader, sending a full tablet report...
I20260620 21:50:37.894591 5334 heartbeater.cc:503] Master 127.0.193.254:42823 was elected leader, sending a full tablet report...
I20260620 21:50:37.894810 5068 heartbeater.cc:503] Master 127.0.193.254:42823 was elected leader, sending a full tablet report...
I20260620 21:50:37.895455 5352 ts_tablet_manager.cc:1428] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 21:50:37.904577 5358 raft_consensus.cc:482] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:37.904752 5358 raft_consensus.cc:504] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ffe45fb8e9443f8b8d5cba801fcbee" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 32813 } } peers { permanent_uuid: "b601e8928a894208922d72e3fb5dd8ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36143 } } peers { permanent_uuid: "522cda7dc47346ab890af10c3c1deeb2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33867 } }
I20260620 21:50:37.905885 5358 leader_election.cc:288] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56ffe45fb8e9443f8b8d5cba801fcbee (127.0.193.193:32813), b601e8928a894208922d72e3fb5dd8ee (127.0.193.194:36143)
I20260620 21:50:37.910776 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.906250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:33639 (local address 127.0.193.193:32813)
0620 21:50:37.906446 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:37.906452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:37.906470 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:50:37.906525 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:37.906528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:37.906599 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:37.906692 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:37.907127 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.907950 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.909546 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.910170 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.910295 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.910367 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:37.910566 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 21:50:37.910647 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:50:37.910781 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.906144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:32813 (local address 127.0.193.195:33639)
0620 21:50:37.906275 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:37.906293 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:37.906385 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:37.906826 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:37.906833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:37.906868 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:37.907012 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.907018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.908056 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.908058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.909411 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.909422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.910287 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.910290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.910427 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.910455 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:50:37.910541 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 21:50:37.910626 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"spinlock_wait_cycles":14592}
I20260620 21:50:37.910833 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.906428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:36143 (local address 127.0.193.195:48621)
0620 21:50:37.906782 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:37.906800 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:37.906917 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:37.907262 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:37.907265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:37.907291 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:37.907385 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.907389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.908292 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.908296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.909595 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:37.909602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.910435 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.910439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:37.910557 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.910576 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 21:50:37.910645 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 21:50:37.910703 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"spinlock_wait_cycles":3584,"thread_start_us":126,"threads_started":1}
I20260620 21:50:37.910851 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.906601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:48621 (local address 127.0.193.194:36143)
0620 21:50:37.906726 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:37.906730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:37.906988 (+ 258us) server_negotiation.cc:407] Connection header received
0620 21:50:37.907050 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:37.907052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:37.907117 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:37.907245 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:37.907502 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.908190 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.909698 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.910282 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.910399 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.910441 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:37.910668 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 21:50:37.910734 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:50:37.911509 5155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c187ed548d5422a98006a251b622da6" candidate_uuid: "522cda7dc47346ab890af10c3c1deeb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b601e8928a894208922d72e3fb5dd8ee" is_pre_election: true
I20260620 21:50:37.911504 5022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c187ed548d5422a98006a251b622da6" candidate_uuid: "522cda7dc47346ab890af10c3c1deeb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ffe45fb8e9443f8b8d5cba801fcbee" is_pre_election: true
I20260620 21:50:37.911808 5022 raft_consensus.cc:2454] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522cda7dc47346ab890af10c3c1deeb2 in term 0.
I20260620 21:50:37.911810 5155 raft_consensus.cc:2454] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522cda7dc47346ab890af10c3c1deeb2 in term 0.
I20260620 21:50:37.912286 5223 leader_election.cc:302] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522cda7dc47346ab890af10c3c1deeb2, b601e8928a894208922d72e3fb5dd8ee; no voters:
I20260620 21:50:37.912606 5358 raft_consensus.cc:2781] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:37.912714 5358 raft_consensus.cc:482] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:37.912782 5358 raft_consensus.cc:3037] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:37.914023 5358 raft_consensus.cc:504] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ffe45fb8e9443f8b8d5cba801fcbee" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 32813 } } peers { permanent_uuid: "b601e8928a894208922d72e3fb5dd8ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36143 } } peers { permanent_uuid: "522cda7dc47346ab890af10c3c1deeb2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33867 } }
I20260620 21:50:37.914336 5358 leader_election.cc:288] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [CANDIDATE]: Term 1 election: Requested vote from peers 56ffe45fb8e9443f8b8d5cba801fcbee (127.0.193.193:32813), b601e8928a894208922d72e3fb5dd8ee (127.0.193.194:36143)
I20260620 21:50:37.914757 5022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c187ed548d5422a98006a251b622da6" candidate_uuid: "522cda7dc47346ab890af10c3c1deeb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ffe45fb8e9443f8b8d5cba801fcbee"
I20260620 21:50:37.914819 5155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c187ed548d5422a98006a251b622da6" candidate_uuid: "522cda7dc47346ab890af10c3c1deeb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b601e8928a894208922d72e3fb5dd8ee"
I20260620 21:50:37.914875 5022 raft_consensus.cc:3037] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:37.914983 5155 raft_consensus.cc:3037] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:37.916046 5022 raft_consensus.cc:2454] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522cda7dc47346ab890af10c3c1deeb2 in term 1.
I20260620 21:50:37.916149 5155 raft_consensus.cc:2454] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522cda7dc47346ab890af10c3c1deeb2 in term 1.
I20260620 21:50:37.916415 5223 leader_election.cc:302] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522cda7dc47346ab890af10c3c1deeb2, 56ffe45fb8e9443f8b8d5cba801fcbee; no voters:
I20260620 21:50:37.916599 5358 raft_consensus.cc:2781] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:37.916882 5358 raft_consensus.cc:686] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [term 1 LEADER]: Becoming Leader. State: Replica: 522cda7dc47346ab890af10c3c1deeb2, State: Running, Role: LEADER
I20260620 21:50:37.917227 5358 consensus_queue.cc:237] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ffe45fb8e9443f8b8d5cba801fcbee" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 32813 } } peers { permanent_uuid: "b601e8928a894208922d72e3fb5dd8ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36143 } } peers { permanent_uuid: "522cda7dc47346ab890af10c3c1deeb2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33867 } }
I20260620 21:50:37.920666 4877 catalog_manager.cc:5495] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 522cda7dc47346ab890af10c3c1deeb2 (127.0.193.195). New cstate: current_term: 1 leader_uuid: "522cda7dc47346ab890af10c3c1deeb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56ffe45fb8e9443f8b8d5cba801fcbee" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 32813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b601e8928a894208922d72e3fb5dd8ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "522cda7dc47346ab890af10c3c1deeb2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33867 } health_report { overall_health: HEALTHY } } }
I20260620 21:50:37.935024 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.931142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.0.193.195:33867)
0620 21:50:37.931339 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:37.931343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:37.931356 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:50:37.931433 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:37.931435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:37.931488 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:37.931581 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:37.931834 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.932393 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.933382 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:37.933602 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:37.933692 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:37.934058 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:37.934504 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:37.934610 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:37.934780 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 21:50:37.934844 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 21:50:37.941403 5022 raft_consensus.cc:1261] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee [term 1 FOLLOWER]: Refusing update from remote peer 522cda7dc47346ab890af10c3c1deeb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:37.941474 5155 raft_consensus.cc:1261] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee [term 1 FOLLOWER]: Refusing update from remote peer 522cda7dc47346ab890af10c3c1deeb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:37.942198 5358 consensus_queue.cc:1035] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ffe45fb8e9443f8b8d5cba801fcbee" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 32813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:37.942405 5361 consensus_queue.cc:1035] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b601e8928a894208922d72e3fb5dd8ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:37.951252 5365 mvcc.cc:205] Tried to move back new op lower bound from 7299040206600212480 to 7299040206511943680. Current Snapshot: MvccSnapshot[applied={T|T < 7299040206600212480}]
I20260620 21:50:37.953341 5366 mvcc.cc:205] Tried to move back new op lower bound from 7299040206600212480 to 7299040206511943680. Current Snapshot: MvccSnapshot[applied={T|T < 7299040206600212480 or (T in {7299040206600212480})}]
I20260620 21:50:37.962028 4876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:37.962236 4876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:37.964812 4876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6f0d56d2a4749acb9af49b43269b05d: Sending DeleteTablet for 3 replicas of tablet 9c187ed548d5422a98006a251b622da6
I20260620 21:50:37.965761 5001 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c187ed548d5422a98006a251b622da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:37 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:48892
I20260620 21:50:37.965817 5135 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c187ed548d5422a98006a251b622da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:37 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:36894
I20260620 21:50:37.966295 5374 tablet_replica.cc:330] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee: stopping tablet replica
I20260620 21:50:37.966612 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 4940
I20260620 21:50:37.966602 5268 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c187ed548d5422a98006a251b622da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:37 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:50408
I20260620 21:50:37.966714 5374 raft_consensus.cc:2229] T 9c187ed548d5422a98006a251b622da6 P 56ffe45fb8e9443f8b8d5cba801fcbee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:37.967345 5376 tablet_replica.cc:330] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2: stopping tablet replica
I20260620 21:50:37.967667 5376 raft_consensus.cc:2229] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:50:37.968149 5376 raft_consensus.cc:2258] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:37.969322 5375 tablet_replica.cc:330] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee: stopping tablet replica
I20260620 21:50:37.969659 5375 raft_consensus.cc:2229] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:37.970098 5375 raft_consensus.cc:2258] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:37.970293 5376 ts_tablet_manager.cc:1905] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:37.973381 5376 ts_tablet_manager.cc:1918] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:37.973495 5376 log.cc:1196] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/9c187ed548d5422a98006a251b622da6
I20260620 21:50:37.973882 5376 ts_tablet_manager.cc:1939] T 9c187ed548d5422a98006a251b622da6 P 522cda7dc47346ab890af10c3c1deeb2: Deleting consensus metadata
I20260620 21:50:37.974660 5375 ts_tablet_manager.cc:1905] T 9c187ed548d5422a98006a251b622da6 P b601e8928a894208922d72e3fb5dd8ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:37.975170 4864 catalog_manager.cc:4841] TS 522cda7dc47346ab890af10c3c1deeb2 (127.0.193.195:33867): tablet 9c187ed548d5422a98006a251b622da6 (table test-table [id=089abb0a90874a7289a515aa7d46901b]) successfully deleted
W20260620 21:50:37.976434 4864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:32813 (error 108)
I20260620 21:50:37.977258 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 5073
I20260620 21:50:37.984229 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.983811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:32813 (local address 127.0.0.1:48896)
0620 21:50:37.984064 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:37.984135 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:32813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
W20260620 21:50:37.984418 4864 catalog_manager.cc:4584] TS 56ffe45fb8e9443f8b8d5cba801fcbee (127.0.193.193:32813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c187ed548d5422a98006a251b622da6: Network error: Client connection negotiation failed: client connection to 127.0.193.193:32813: connect: Connection refused (error 111)
I20260620 21:50:37.985245 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 5206
I20260620 21:50:37.990774 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:37.990435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:36143 (local address 127.0.0.1:36902)
0620 21:50:37.990586 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:37.990656 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.194:36143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 21:50:37.990993 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 4847
2026-06-20T21:50:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4217 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T21:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:38Z Disabled control of system clock
I20260620 21:50:38.023564 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:45249
--webserver_interface=127.0.193.254
--webserver_port=0
--builtin_ntp_servers=127.0.193.212:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:45249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 21:50:38.136981 5381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:38.137257 5381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:38.137296 5381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:38.141563 5381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:38.141640 5381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:38.141657 5381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:38.141675 5381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:38.141690 5381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:38.146638 5381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:45249
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:45249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:38.147907 5381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:38.149333 5381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:38.158041 5389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:38.158046 5386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:38.158046 5387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:38.158087 5381 server_base.cc:1034] running on GCE node
I20260620 21:50:38.159016 5381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:38.160264 5381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:38.161442 5381 hybrid_clock.cc:648] HybridClock initialized: now 1781992238161424 us; error 36 us; skew 500 ppm
I20260620 21:50:38.163477 5381 webserver.cc:460] Webserver started at http://127.0.193.254:46851/ using document root <none> and password file <none>
I20260620 21:50:38.164077 5381 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:38.164143 5381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:38.164362 5381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:38.166469 5381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "55d5994d57644f2fbe13396a3f9cb618"
format_stamp: "Formatted at 2026-06-20 21:50:38 on dist-test-slave-kk5c"
server_key: "3068f16f639ab50346e909e8150d9478"
server_key_iv: "5413302e55f3786805bc885c8bc5d879"
server_key_version: "encryptionkey@0"
I20260620 21:50:38.167085 5381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "55d5994d57644f2fbe13396a3f9cb618"
format_stamp: "Formatted at 2026-06-20 21:50:38 on dist-test-slave-kk5c"
server_key: "3068f16f639ab50346e909e8150d9478"
server_key_iv: "5413302e55f3786805bc885c8bc5d879"
server_key_version: "encryptionkey@0"
I20260620 21:50:38.171156 5381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:50:38.173902 5395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:38.175514 5381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:38.175662 5381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "55d5994d57644f2fbe13396a3f9cb618"
format_stamp: "Formatted at 2026-06-20 21:50:38 on dist-test-slave-kk5c"
server_key: "3068f16f639ab50346e909e8150d9478"
server_key_iv: "5413302e55f3786805bc885c8bc5d879"
server_key_version: "encryptionkey@0"
I20260620 21:50:38.175794 5381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:38.180642 5381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:38.181299 5381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:38.181493 5381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:38.189002 5381 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:45249
I20260620 21:50:38.189049 5447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:45249 every 8 connection(s)
I20260620 21:50:38.189996 5381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:38.193511 5448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:38.199520 5448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618: Bootstrap starting.
I20260620 21:50:38.199837 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 5381
I20260620 21:50:38.199971 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:38.200234 775 external_mini_cluster.cc:1428] Setting key 1a42db4549b09f296cc323c23f27be52
I20260620 21:50:38.203022 5448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:38.203950 5448 log.cc:824] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:38.206097 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:38.201670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.0.193.254:45249)
0620 21:50:38.202260 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:38.202279 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:38.202314 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:50:38.202962 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:38.202979 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:38.203454 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:38.203733 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:38.204649 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:38.204654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:38.204671 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:38.204686 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:38.204755 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:38.204780 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:38.204928 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:38.205201 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 21:50:38.205414 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":117,"threads_started":1}
I20260620 21:50:38.206527 5448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618: No bootstrap required, opened a new log
I20260620 21:50:38.210870 5448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d5994d57644f2fbe13396a3f9cb618" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45249 } }
I20260620 21:50:38.211205 5448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:38.211263 5448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d5994d57644f2fbe13396a3f9cb618, State: Initialized, Role: FOLLOWER
I20260620 21:50:38.211771 5448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d5994d57644f2fbe13396a3f9cb618" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45249 } }
I20260620 21:50:38.211920 5448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:38.211973 5448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:38.212049 5448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:38.213090 5448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d5994d57644f2fbe13396a3f9cb618" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45249 } }
I20260620 21:50:38.213408 5448 leader_election.cc:302] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55d5994d57644f2fbe13396a3f9cb618; no voters:
I20260620 21:50:38.213687 5448 leader_election.cc:288] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:38.213828 5453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:38.214052 5453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [term 1 LEADER]: Becoming Leader. State: Replica: 55d5994d57644f2fbe13396a3f9cb618, State: Running, Role: LEADER
I20260620 21:50:38.214437 5453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d5994d57644f2fbe13396a3f9cb618" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45249 } }
I20260620 21:50:38.215063 5448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:38.216213 5454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55d5994d57644f2fbe13396a3f9cb618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d5994d57644f2fbe13396a3f9cb618" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45249 } } }
I20260620 21:50:38.216362 5454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:38.216213 5455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55d5994d57644f2fbe13396a3f9cb618. Latest consensus state: current_term: 1 leader_uuid: "55d5994d57644f2fbe13396a3f9cb618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d5994d57644f2fbe13396a3f9cb618" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45249 } } }
I20260620 21:50:38.216466 5455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:38.216841 5462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:38.220016 5462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:38.225260 5462 catalog_manager.cc:1347] Generated new cluster ID: bd2b13168ae54a23bf544e5331ef4c62
I20260620 21:50:38.225349 5462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:38.238431 5462 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:38.238795 5462 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:38.249471 5462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618: Generated new TSK 0
I20260620 21:50:38.250201 5462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:50:38.262910 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:45249
--builtin_ntp_servers=127.0.193.212:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 21:50:38.398633 5472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:38.399013 5472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:38.399094 5472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:38.403642 5472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:38.403760 5472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:38.404155 5472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:38.410069 5472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:45249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:38.411324 5472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:38.412721 5472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:50:38.423509 5472 server_base.cc:1034] running on GCE node
W20260620 21:50:38.423453 5480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:38.423478 5478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:38.423453 5477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:38.424106 5472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:38.424830 5472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:38.426020 5472 hybrid_clock.cc:648] HybridClock initialized: now 1781992238426000 us; error 37 us; skew 500 ppm
I20260620 21:50:38.428129 5472 webserver.cc:460] Webserver started at http://127.0.193.193:41663/ using document root <none> and password file <none>
I20260620 21:50:38.428786 5472 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:38.428884 5472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:38.429124 5472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:38.431401 5472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "c94399cd850a49da9800525fc23f2b82"
format_stamp: "Formatted at 2026-06-20 21:50:38 on dist-test-slave-kk5c"
server_key: "87d4833b23d128b9cdcb3804faa63883"
server_key_iv: "3a4117e9ae9aeb2aa23cd4fab660d358"
server_key_version: "encryptionkey@0"
I20260620 21:50:38.432024 5472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "c94399cd850a49da9800525fc23f2b82"
format_stamp: "Formatted at 2026-06-20 21:50:38 on dist-test-slave-kk5c"
server_key: "87d4833b23d128b9cdcb3804faa63883"
server_key_iv: "3a4117e9ae9aeb2aa23cd4fab660d358"
server_key_version: "encryptionkey@0"
I20260620 21:50:38.435932 5472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:50:38.438627 5486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:38.440193 5472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:38.440403 5472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "c94399cd850a49da9800525fc23f2b82"
format_stamp: "Formatted at 2026-06-20 21:50:38 on dist-test-slave-kk5c"
server_key: "87d4833b23d128b9cdcb3804faa63883"
server_key_iv: "3a4117e9ae9aeb2aa23cd4fab660d358"
server_key_version: "encryptionkey@0"
I20260620 21:50:38.440544 5472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:38.445286 5472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:38.445961 5472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:38.446166 5472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:38.446764 5472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:38.447810 5472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:38.447885 5472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:38.447974 5472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:38.448028 5472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:38.457309 5472 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:35569
I20260620 21:50:38.457392 5599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:35569 every 8 connection(s)
I20260620 21:50:38.458353 5472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:38.460696 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 5472
I20260620 21:50:38.460839 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:50:38.461159 775 external_mini_cluster.cc:1428] Setting key adfea91109fb0293e7e1122ed08c12a9
I20260620 21:50:38.463519 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:45249
--builtin_ntp_servers=127.0.193.212:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 21:50:38.465834 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:38.460555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:48831 (local address 127.0.193.254:45249)
0620 21:50:38.460737 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:38.460742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:38.462003 (+ 1261us) server_negotiation.cc:407] Connection header received
0620 21:50:38.463764 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:38.463770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:38.463819 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:38.463919 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:38.465251 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:38.465256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:38.465261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:38.465278 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:38.465304 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:38.465316 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:38.465490 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:38.465618 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 21:50:38.465670 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 21:50:38.466840 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:38.461047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:45249 (local address 127.0.193.193:48831)
0620 21:50:38.461858 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:38.461916 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:38.463491 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:38.464132 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:38.464145 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:38.464269 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:38.464913 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:50:38.464917 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:38.464982 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:50:38.465078 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:38.465477 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:38.465484 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:38.465488 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:50:38.465626 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 21:50:38.465830 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":161,"threads_started":1}
I20260620 21:50:38.468209 5600 heartbeater.cc:347] Connected to a master server at 127.0.193.254:45249
I20260620 21:50:38.468528 5600 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:38.469014 5600 heartbeater.cc:511] Master 127.0.193.254:45249 requested a full tablet report, sending...
I20260620 21:50:38.470717 5412 ts_manager.cc:194] Registered new tserver with Master: c94399cd850a49da9800525fc23f2b82 (127.0.193.193:35569)
W20260620 21:50:38.588575 5603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:38.588914 5603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:38.589000 5603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:38.593250 5603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:38.593549 5603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:38.593711 5603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:38.598867 5603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:45249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:38.600099 5603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:38.601559 5603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:38.611443 5608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:38.611390 5609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:38.611600 5603 server_base.cc:1034] running on GCE node
W20260620 21:50:38.611413 5611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:38.612172 5603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:38.612922 5603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:38.614143 5603 hybrid_clock.cc:648] HybridClock initialized: now 1781992238614125 us; error 32 us; skew 500 ppm
I20260620 21:50:38.616483 5603 webserver.cc:460] Webserver started at http://127.0.193.194:43443/ using document root <none> and password file <none>
I20260620 21:50:38.617184 5603 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:38.617249 5603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:38.617487 5603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:38.619684 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "1a5228f9a372489ea201274e4b68f324"
format_stamp: "Formatted at 2026-06-20 21:50:38 on dist-test-slave-kk5c"
server_key: "d673626d2814a1e8ed25abf93f514c9a"
server_key_iv: "0b02068b7d76d9255b1e86747a3cc1a2"
server_key_version: "encryptionkey@0"
I20260620 21:50:38.620270 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "1a5228f9a372489ea201274e4b68f324"
format_stamp: "Formatted at 2026-06-20 21:50:38 on dist-test-slave-kk5c"
server_key: "d673626d2814a1e8ed25abf93f514c9a"
server_key_iv: "0b02068b7d76d9255b1e86747a3cc1a2"
server_key_version: "encryptionkey@0"
I20260620 21:50:38.624313 5603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:50:38.626981 5617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:38.628499 5603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:38.628664 5603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "1a5228f9a372489ea201274e4b68f324"
format_stamp: "Formatted at 2026-06-20 21:50:38 on dist-test-slave-kk5c"
server_key: "d673626d2814a1e8ed25abf93f514c9a"
server_key_iv: "0b02068b7d76d9255b1e86747a3cc1a2"
server_key_version: "encryptionkey@0"
I20260620 21:50:38.628799 5603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:38.633742 5603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:38.634413 5603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:38.634613 5603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:38.635246 5603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:38.636193 5603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:38.636264 5603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:38.636334 5603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:38.636385 5603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:38.645515 5730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:35239 every 8 connection(s)
I20260620 21:50:38.645493 5603 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:35239
I20260620 21:50:38.646548 5603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:38.653355 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:38.648634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:45607 (local address 127.0.193.254:45249)
0620 21:50:38.648862 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:38.648867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:38.650032 (+ 1165us) server_negotiation.cc:407] Connection header received
0620 21:50:38.651497 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:38.651500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:38.651543 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:38.651630 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:38.652835 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:38.652838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:38.652840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:38.652853 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:38.652872 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:38.652880 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:38.652974 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:38.653177 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 21:50:38.653218 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:50:38.654301 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:38.649157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:45249 (local address 127.0.193.194:45607)
0620 21:50:38.649885 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:38.649930 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:38.651122 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:38.651817 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:38.651825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:38.651942 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:38.652528 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:50:38.652530 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:38.652570 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:50:38.652640 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:38.653022 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:38.653030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:38.653035 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:50:38.653194 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 21:50:38.653400 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":141,"threads_started":1}
I20260620 21:50:38.655344 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 5603
I20260620 21:50:38.655525 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:50:38.655719 5731 heartbeater.cc:347] Connected to a master server at 127.0.193.254:45249
I20260620 21:50:38.655885 775 external_mini_cluster.cc:1428] Setting key fc594847023e8bc2c70f81d3157b66b0
I20260620 21:50:38.655988 5731 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:38.656490 5731 heartbeater.cc:511] Master 127.0.193.254:45249 requested a full tablet report, sending...
I20260620 21:50:38.657703 5412 ts_manager.cc:194] Registered new tserver with Master: 1a5228f9a372489ea201274e4b68f324 (127.0.193.194:35239)
I20260620 21:50:38.658077 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:45249
--builtin_ntp_servers=127.0.193.212:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 21:50:38.782464 5734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:38.782775 5734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:38.782817 5734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:38.787350 5734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:38.787439 5734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:38.787577 5734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:38.794369 5734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:45249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:38.795614 5734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:38.797088 5734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:38.808317 5739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:38.808350 5734 server_base.cc:1034] running on GCE node
W20260620 21:50:38.808212 5742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:38.808216 5740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:38.808925 5734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:38.809587 5734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:38.810786 5734 hybrid_clock.cc:648] HybridClock initialized: now 1781992238810766 us; error 34 us; skew 500 ppm
I20260620 21:50:38.812911 5734 webserver.cc:460] Webserver started at http://127.0.193.195:38343/ using document root <none> and password file <none>
I20260620 21:50:38.813560 5734 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:38.813848 5734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:38.814157 5734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:38.816875 5734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "39b50f9fb31c439f9e740a90358793d6"
format_stamp: "Formatted at 2026-06-20 21:50:38 on dist-test-slave-kk5c"
server_key: "9c7f54e251edec9b966df13fc88b6b03"
server_key_iv: "cf305c6fd835a47d82ecf76044a2f97d"
server_key_version: "encryptionkey@0"
I20260620 21:50:38.817497 5734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "39b50f9fb31c439f9e740a90358793d6"
format_stamp: "Formatted at 2026-06-20 21:50:38 on dist-test-slave-kk5c"
server_key: "9c7f54e251edec9b966df13fc88b6b03"
server_key_iv: "cf305c6fd835a47d82ecf76044a2f97d"
server_key_version: "encryptionkey@0"
I20260620 21:50:38.821612 5734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:38.824849 5748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:38.826617 5734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:50:38.826759 5734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "39b50f9fb31c439f9e740a90358793d6"
format_stamp: "Formatted at 2026-06-20 21:50:38 on dist-test-slave-kk5c"
server_key: "9c7f54e251edec9b966df13fc88b6b03"
server_key_iv: "cf305c6fd835a47d82ecf76044a2f97d"
server_key_version: "encryptionkey@0"
I20260620 21:50:38.826871 5734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:38.831866 5734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:38.832717 5734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:38.832947 5734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:38.833590 5734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:38.834523 5734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:38.834573 5734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:38.834612 5734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:38.834628 5734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:38.844669 5861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:37831 every 8 connection(s)
I20260620 21:50:38.844704 5734 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:37831
I20260620 21:50:38.845923 5734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:50:38.852947 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:38.848134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:39371 (local address 127.0.193.254:45249)
0620 21:50:38.848296 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:38.848300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:38.849609 (+ 1309us) server_negotiation.cc:407] Connection header received
0620 21:50:38.851038 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:38.851042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:38.851090 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:38.851166 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:38.852424 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:38.852432 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:38.852434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:38.852454 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:38.852475 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:38.852483 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:38.852589 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:38.852747 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 21:50:38.852797 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 21:50:38.854185 5864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:38.848524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:45249 (local address 127.0.193.195:39371)
0620 21:50:38.849290 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:38.849459 (+ 169us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:38.850794 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:38.851385 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:38.851393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:38.851494 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:38.852123 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:50:38.852125 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:38.852168 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:50:38.852242 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:38.852580 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:38.852588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:38.852595 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:50:38.852797 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 21:50:38.853076 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":119,"threads_started":1}
I20260620 21:50:38.855489 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 5734
I20260620 21:50:38.855615 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:50:38.855729 5862 heartbeater.cc:347] Connected to a master server at 127.0.193.254:45249
I20260620 21:50:38.856096 5862 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:38.856112 775 external_mini_cluster.cc:1428] Setting key b6557ec87bc7c6b1bc47db15e2a14129
I20260620 21:50:38.856603 5862 heartbeater.cc:511] Master 127.0.193.254:45249 requested a full tablet report, sending...
I20260620 21:50:38.857919 5412 ts_manager.cc:194] Registered new tserver with Master: 39b50f9fb31c439f9e740a90358793d6 (127.0.193.195:37831)
I20260620 21:50:38.871402 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:50:38.876024 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:38.873814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.0.193.254:45249)
0620 21:50:38.874082 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:38.874086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:38.874151 (+ 65us) server_negotiation.cc:407] Connection header received
0620 21:50:38.874290 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:38.874293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:38.874333 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:38.874415 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:38.875478 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:38.875482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:38.875484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:38.875495 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:38.875517 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:38.875525 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:38.875612 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:38.875822 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 21:50:38.875866 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 21:50:38.879899 5412 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:50:38.883126 5412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:38.896313 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:38.892966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:37831 (local address 127.0.0.1:35772)
0620 21:50:38.893406 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:38.893464 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:38.893570 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:38.894665 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:38.894673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:38.894768 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:38.895477 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:50:38.895480 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:38.895513 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:50:38.895591 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:38.895926 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:38.895934 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:38.895940 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:50:38.896050 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 21:50:38.896123 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":85,"threads_started":1}
I20260620 21:50:38.896699 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:38.893257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35772 (local address 127.0.193.195:37831)
0620 21:50:38.893748 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:38.893760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:38.893784 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:50:38.893841 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:38.893846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:38.894052 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:38.894213 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:38.895738 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:38.895745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:38.895757 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:38.895769 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:38.895816 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:38.895838 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:38.895958 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:38.896189 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 21:50:38.896267 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":162,"threads_started":1}
I20260620 21:50:38.896894 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:38.893729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:35239 (local address 127.0.0.1:40776)
0620 21:50:38.894608 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:38.894620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:38.894728 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:38.895161 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:38.895164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:38.895179 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:38.895604 (+ 425us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:50:38.895606 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:38.895617 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:50:38.895643 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:38.896395 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:38.896398 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:38.896401 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:50:38.896465 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 21:50:38.896499 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":91,"threads_started":1}
I20260620 21:50:38.897182 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:38.893005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:35569 (local address 127.0.0.1:54076)
0620 21:50:38.894422 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:38.894439 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:38.894521 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:38.896029 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:38.896034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:38.896049 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:38.896495 (+ 446us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:50:38.896497 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:38.896507 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:50:38.896536 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:38.896955 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:38.896959 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:38.896964 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:50:38.897037 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 21:50:38.897080 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1349,"thread_start_us":96,"threads_started":1}
I20260620 21:50:38.897341 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:38.893887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.0.193.194:35239)
0620 21:50:38.894286 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:38.894297 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:38.894721 (+ 424us) server_negotiation.cc:407] Connection header received
0620 21:50:38.894818 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:38.894825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:38.895008 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:38.895450 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:38.896191 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:38.896197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:38.896207 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:38.896217 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:38.896252 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:38.896274 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:38.897071 (+ 797us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:38.897115 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 21:50:38.897182 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":116,"threads_started":1}
I20260620 21:50:38.897622 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:38.893936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.0.193.193:35569)
0620 21:50:38.895019 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:38.895031 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:38.895057 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:50:38.895112 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:38.895120 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:38.895369 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:38.895491 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:38.896671 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:38.896676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:38.896687 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:38.896698 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:38.896771 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:38.896810 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:38.896913 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:38.897374 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 21:50:38.897455 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":129,"threads_started":1}
I20260620 21:50:38.899974 5796 tablet_service.cc:1467] Processing CreateTablet for tablet 9ecb0689b8234283b5c41259952451ae (DEFAULT_TABLE table=test-table [id=f82fdc6ac42a4da8abecaddb494f6e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:38.900226 5665 tablet_service.cc:1467] Processing CreateTablet for tablet 9ecb0689b8234283b5c41259952451ae (DEFAULT_TABLE table=test-table [id=f82fdc6ac42a4da8abecaddb494f6e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:38.900738 5534 tablet_service.cc:1467] Processing CreateTablet for tablet 9ecb0689b8234283b5c41259952451ae (DEFAULT_TABLE table=test-table [id=f82fdc6ac42a4da8abecaddb494f6e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:38.901173 5796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecb0689b8234283b5c41259952451ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:38.901194 5665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecb0689b8234283b5c41259952451ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:38.901662 5534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecb0689b8234283b5c41259952451ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:38.908262 5879 tablet_bootstrap.cc:492] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324: Bootstrap starting.
I20260620 21:50:38.909312 5881 tablet_bootstrap.cc:492] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6: Bootstrap starting.
I20260620 21:50:38.911607 5880 tablet_bootstrap.cc:492] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82: Bootstrap starting.
I20260620 21:50:38.912020 5881 tablet_bootstrap.cc:654] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:38.912026 5879 tablet_bootstrap.cc:654] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:38.912973 5881 log.cc:824] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:38.913018 5879 log.cc:824] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:38.914206 5880 tablet_bootstrap.cc:654] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:38.915140 5880 log.cc:824] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:38.915511 5879 tablet_bootstrap.cc:492] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324: No bootstrap required, opened a new log
I20260620 21:50:38.915730 5879 ts_tablet_manager.cc:1397] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 21:50:38.915753 5881 tablet_bootstrap.cc:492] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6: No bootstrap required, opened a new log
I20260620 21:50:38.915968 5881 ts_tablet_manager.cc:1397] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 21:50:38.918241 5880 tablet_bootstrap.cc:492] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82: No bootstrap required, opened a new log
I20260620 21:50:38.918467 5880 ts_tablet_manager.cc:1397] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 21:50:38.919746 5879 raft_consensus.cc:348] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b50f9fb31c439f9e740a90358793d6" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37831 } } peers { permanent_uuid: "c94399cd850a49da9800525fc23f2b82" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35569 } } peers { permanent_uuid: "1a5228f9a372489ea201274e4b68f324" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35239 } }
I20260620 21:50:38.920007 5879 raft_consensus.cc:374] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:38.920063 5879 raft_consensus.cc:729] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a5228f9a372489ea201274e4b68f324, State: Initialized, Role: FOLLOWER
I20260620 21:50:38.920534 5879 consensus_queue.cc:260] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b50f9fb31c439f9e740a90358793d6" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37831 } } peers { permanent_uuid: "c94399cd850a49da9800525fc23f2b82" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35569 } } peers { permanent_uuid: "1a5228f9a372489ea201274e4b68f324" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35239 } }
I20260620 21:50:38.921264 5731 heartbeater.cc:503] Master 127.0.193.254:45249 was elected leader, sending a full tablet report...
I20260620 21:50:38.921615 5879 ts_tablet_manager.cc:1428] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:50:38.920944 5881 raft_consensus.cc:348] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b50f9fb31c439f9e740a90358793d6" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37831 } } peers { permanent_uuid: "c94399cd850a49da9800525fc23f2b82" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35569 } } peers { permanent_uuid: "1a5228f9a372489ea201274e4b68f324" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35239 } }
I20260620 21:50:38.922623 5881 raft_consensus.cc:374] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:38.922695 5881 raft_consensus.cc:729] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b50f9fb31c439f9e740a90358793d6, State: Initialized, Role: FOLLOWER
I20260620 21:50:38.923317 5881 consensus_queue.cc:260] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b50f9fb31c439f9e740a90358793d6" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37831 } } peers { permanent_uuid: "c94399cd850a49da9800525fc23f2b82" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35569 } } peers { permanent_uuid: "1a5228f9a372489ea201274e4b68f324" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35239 } }
I20260620 21:50:38.923614 5880 raft_consensus.cc:348] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b50f9fb31c439f9e740a90358793d6" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37831 } } peers { permanent_uuid: "c94399cd850a49da9800525fc23f2b82" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35569 } } peers { permanent_uuid: "1a5228f9a372489ea201274e4b68f324" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35239 } }
I20260620 21:50:38.923897 5880 raft_consensus.cc:374] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:38.923946 5880 raft_consensus.cc:729] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94399cd850a49da9800525fc23f2b82, State: Initialized, Role: FOLLOWER
I20260620 21:50:38.924408 5880 consensus_queue.cc:260] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b50f9fb31c439f9e740a90358793d6" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37831 } } peers { permanent_uuid: "c94399cd850a49da9800525fc23f2b82" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35569 } } peers { permanent_uuid: "1a5228f9a372489ea201274e4b68f324" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35239 } }
I20260620 21:50:38.924492 5881 ts_tablet_manager.cc:1428] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260620 21:50:38.924887 5862 heartbeater.cc:503] Master 127.0.193.254:45249 was elected leader, sending a full tablet report...
I20260620 21:50:38.925216 5600 heartbeater.cc:503] Master 127.0.193.254:45249 was elected leader, sending a full tablet report...
I20260620 21:50:38.925415 5880 ts_tablet_manager.cc:1428] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260620 21:50:38.959654 5601 tablet.cc:2369] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:50:39.097175 5863 tablet.cc:2369] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:50:39.148211 5732 tablet.cc:2369] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:50:39.190295 5886 raft_consensus.cc:482] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:39.190536 5886 raft_consensus.cc:504] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b50f9fb31c439f9e740a90358793d6" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37831 } } peers { permanent_uuid: "c94399cd850a49da9800525fc23f2b82" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35569 } } peers { permanent_uuid: "1a5228f9a372489ea201274e4b68f324" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35239 } }
I20260620 21:50:39.192113 5886 leader_election.cc:288] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c94399cd850a49da9800525fc23f2b82 (127.0.193.193:35569), 1a5228f9a372489ea201274e4b68f324 (127.0.193.194:35239)
I20260620 21:50:39.194574 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:39.192513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:47541 (local address 127.0.193.193:35569)
0620 21:50:39.192650 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:39.192654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:39.192668 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:50:39.192817 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:39.192820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:39.192864 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:39.192994 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:39.194171 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:39.194174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:39.194176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:39.194187 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:39.194205 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:39.194213 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:39.194323 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:39.194393 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 21:50:39.194436 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:50:39.194574 5864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:39.192403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:35569 (local address 127.0.193.195:47541)
0620 21:50:39.192550 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:39.192567 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:39.192669 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:39.193104 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:39.193108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:39.193123 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:39.193802 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:50:39.193803 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:39.193816 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:50:39.193838 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:39.194307 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:39.194310 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:39.194313 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 21:50:39.194426 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 21:50:39.194458 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 21:50:39.194792 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:39.193030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:46217 (local address 127.0.193.194:35239)
0620 21:50:39.193156 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:39.193163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:39.193453 (+ 290us) server_negotiation.cc:407] Connection header received
0620 21:50:39.193556 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:39.193560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:39.193604 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:39.193702 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:39.194351 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:39.194355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:39.194357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:39.194369 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:39.194389 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:39.194398 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:39.194459 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:39.194633 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 21:50:39.194674 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:50:39.195099 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:39.192627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:35239 (local address 127.0.193.195:46217)
0620 21:50:39.193261 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:39.193348 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:39.193462 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:39.193745 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:39.193748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:39.193771 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:39.194173 (+ 402us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:50:39.194175 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:39.194184 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:50:39.194207 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:39.194532 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:39.194536 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:39.194538 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 21:50:39.194945 (+ 407us) client_negotiation.cc:240] Negotiation successful
0620 21:50:39.194980 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":114,"threads_started":1}
I20260620 21:50:39.195271 5554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecb0689b8234283b5c41259952451ae" candidate_uuid: "39b50f9fb31c439f9e740a90358793d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94399cd850a49da9800525fc23f2b82" is_pre_election: true
I20260620 21:50:39.195637 5554 raft_consensus.cc:2454] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b50f9fb31c439f9e740a90358793d6 in term 0.
I20260620 21:50:39.195910 5685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecb0689b8234283b5c41259952451ae" candidate_uuid: "39b50f9fb31c439f9e740a90358793d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5228f9a372489ea201274e4b68f324" is_pre_election: true
I20260620 21:50:39.196152 5750 leader_election.cc:302] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b50f9fb31c439f9e740a90358793d6, c94399cd850a49da9800525fc23f2b82; no voters:
I20260620 21:50:39.196210 5685 raft_consensus.cc:2454] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b50f9fb31c439f9e740a90358793d6 in term 0.
I20260620 21:50:39.196475 5886 raft_consensus.cc:2781] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:39.196558 5886 raft_consensus.cc:482] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:39.196607 5886 raft_consensus.cc:3037] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:39.197629 5886 raft_consensus.cc:504] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b50f9fb31c439f9e740a90358793d6" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37831 } } peers { permanent_uuid: "c94399cd850a49da9800525fc23f2b82" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35569 } } peers { permanent_uuid: "1a5228f9a372489ea201274e4b68f324" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35239 } }
I20260620 21:50:39.198017 5886 leader_election.cc:288] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [CANDIDATE]: Term 1 election: Requested vote from peers c94399cd850a49da9800525fc23f2b82 (127.0.193.193:35569), 1a5228f9a372489ea201274e4b68f324 (127.0.193.194:35239)
I20260620 21:50:39.198390 5554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecb0689b8234283b5c41259952451ae" candidate_uuid: "39b50f9fb31c439f9e740a90358793d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94399cd850a49da9800525fc23f2b82"
I20260620 21:50:39.198398 5685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecb0689b8234283b5c41259952451ae" candidate_uuid: "39b50f9fb31c439f9e740a90358793d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5228f9a372489ea201274e4b68f324"
I20260620 21:50:39.198518 5685 raft_consensus.cc:3037] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:39.198519 5554 raft_consensus.cc:3037] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:39.200030 5554 raft_consensus.cc:2454] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b50f9fb31c439f9e740a90358793d6 in term 1.
I20260620 21:50:39.200008 5685 raft_consensus.cc:2454] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b50f9fb31c439f9e740a90358793d6 in term 1.
I20260620 21:50:39.200364 5750 leader_election.cc:302] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b50f9fb31c439f9e740a90358793d6, c94399cd850a49da9800525fc23f2b82; no voters:
I20260620 21:50:39.200594 5886 raft_consensus.cc:2781] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:39.200906 5886 raft_consensus.cc:686] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [term 1 LEADER]: Becoming Leader. State: Replica: 39b50f9fb31c439f9e740a90358793d6, State: Running, Role: LEADER
I20260620 21:50:39.201308 5886 consensus_queue.cc:237] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b50f9fb31c439f9e740a90358793d6" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37831 } } peers { permanent_uuid: "c94399cd850a49da9800525fc23f2b82" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35569 } } peers { permanent_uuid: "1a5228f9a372489ea201274e4b68f324" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35239 } }
I20260620 21:50:39.204967 5411 catalog_manager.cc:5495] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39b50f9fb31c439f9e740a90358793d6 (127.0.193.195). New cstate: current_term: 1 leader_uuid: "39b50f9fb31c439f9e740a90358793d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b50f9fb31c439f9e740a90358793d6" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c94399cd850a49da9800525fc23f2b82" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a5228f9a372489ea201274e4b68f324" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35239 } health_report { overall_health: UNKNOWN } } }
I20260620 21:50:39.265034 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:39.263321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35778 (local address 127.0.193.195:37831)
0620 21:50:39.263530 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:39.263534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:39.263546 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:50:39.263619 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:39.263621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:39.263676 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:39.263779 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:39.264605 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:39.264613 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:39.264615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:39.264626 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:39.264643 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:39.264650 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:39.264741 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:39.264838 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 21:50:39.264883 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 21:50:39.272097 5554 raft_consensus.cc:1261] T 9ecb0689b8234283b5c41259952451ae P c94399cd850a49da9800525fc23f2b82 [term 1 FOLLOWER]: Refusing update from remote peer 39b50f9fb31c439f9e740a90358793d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:39.272097 5685 raft_consensus.cc:1261] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324 [term 1 FOLLOWER]: Refusing update from remote peer 39b50f9fb31c439f9e740a90358793d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:39.273083 5889 consensus_queue.cc:1035] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a5228f9a372489ea201274e4b68f324" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:39.273310 5886 consensus_queue.cc:1035] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c94399cd850a49da9800525fc23f2b82" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:39.284519 5894 mvcc.cc:205] Tried to move back new op lower bound from 7299040212049059840 to 7299040211771817984. Current Snapshot: MvccSnapshot[applied={T|T < 7299040212049059840}]
I20260620 21:50:39.291704 5411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:39.291862 5411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:39.294445 5411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55d5994d57644f2fbe13396a3f9cb618: Sending DeleteTablet for 3 replicas of tablet 9ecb0689b8234283b5c41259952451ae
I20260620 21:50:39.295372 5796 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ecb0689b8234283b5c41259952451ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:39 UTC) from {username='slave'} at 127.0.0.1:35772
I20260620 21:50:39.295462 5665 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ecb0689b8234283b5c41259952451ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:39 UTC) from {username='slave'} at 127.0.0.1:40776
I20260620 21:50:39.295950 5534 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ecb0689b8234283b5c41259952451ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:39 UTC) from {username='slave'} at 127.0.0.1:54076
I20260620 21:50:39.296170 5902 tablet_replica.cc:330] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6: stopping tablet replica
I20260620 21:50:39.296518 5902 raft_consensus.cc:2229] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:50:39.295964 5903 tablet_replica.cc:330] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324: stopping tablet replica
I20260620 21:50:39.296928 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 5472
I20260620 21:50:39.296929 5903 raft_consensus.cc:2229] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:39.296957 5902 raft_consensus.cc:2258] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:39.297593 5903 raft_consensus.cc:2258] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:39.299865 5902 ts_tablet_manager.cc:1905] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:39.300629 5903 ts_tablet_manager.cc:1905] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:39.302851 5902 ts_tablet_manager.cc:1918] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:39.303045 5902 log.cc:1196] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/9ecb0689b8234283b5c41259952451ae
I20260620 21:50:39.303277 5903 ts_tablet_manager.cc:1918] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:39.303390 5903 log.cc:1196] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/9ecb0689b8234283b5c41259952451ae
I20260620 21:50:39.303653 5902 ts_tablet_manager.cc:1939] T 9ecb0689b8234283b5c41259952451ae P 39b50f9fb31c439f9e740a90358793d6: Deleting consensus metadata
I20260620 21:50:39.303709 5903 ts_tablet_manager.cc:1939] T 9ecb0689b8234283b5c41259952451ae P 1a5228f9a372489ea201274e4b68f324: Deleting consensus metadata
I20260620 21:50:39.305083 5397 catalog_manager.cc:4841] TS 1a5228f9a372489ea201274e4b68f324 (127.0.193.194:35239): tablet 9ecb0689b8234283b5c41259952451ae (table test-table [id=f82fdc6ac42a4da8abecaddb494f6e84]) successfully deleted
I20260620 21:50:39.305084 5396 catalog_manager.cc:4841] TS 39b50f9fb31c439f9e740a90358793d6 (127.0.193.195:37831): tablet 9ecb0689b8234283b5c41259952451ae (table test-table [id=f82fdc6ac42a4da8abecaddb494f6e84]) successfully deleted
W20260620 21:50:39.306963 5397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:35569 (error 108)
I20260620 21:50:39.307103 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 5603
I20260620 21:50:39.308287 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:39.307888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:35569 (local address 127.0.0.1:54092)
0620 21:50:39.308092 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:39.308168 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260620 21:50:39.308456 5397 catalog_manager.cc:4584] TS c94399cd850a49da9800525fc23f2b82 (127.0.193.193:35569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ecb0689b8234283b5c41259952451ae: Network error: Client connection negotiation failed: client connection to 127.0.193.193:35569: connect: Connection refused (error 111)
I20260620 21:50:39.313426 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 5734
I20260620 21:50:39.319269 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 5381
2026-06-20T21:50:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1326 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 21:50:39.341512 775 mini_oidc.cc:152] Starting JWKS server
I20260620 21:50:39.343053 775 webserver.cc:460] Webserver started at https://127.0.0.1:41805/ using document root <none> and password file <none>
I20260620 21:50:39.343384 775 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 21:50:39.343633 775 webserver.cc:460] Webserver started at http://127.0.0.1:34109/ using document root <none> and password file <none>
2026-06-20T21:50:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:39Z Disabled control of system clock
I20260620 21:50:39.353082 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:39685
--webserver_interface=127.0.193.254
--webserver_port=0
--builtin_ntp_servers=127.0.193.212:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:39685
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41805/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/testchainca.cert with env {}
W20260620 21:50:39.470782 5911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:39.471084 5911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:39.471127 5911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:39.475385 5911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:39.475461 5911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:39.475477 5911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:39.475494 5911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:39.475510 5911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:50:39.475526 5911 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:50:39.475540 5911 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41805/jwks.json
I20260620 21:50:39.481467 5911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:39685
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:39685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41805/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:39.482551 5911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:39.483572 5911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:39.492188 5919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:39.492197 5916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:39.492185 5917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:39.492599 5911 server_base.cc:1034] running on GCE node
I20260620 21:50:39.493328 5911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:39.494501 5911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:39.495731 5911 hybrid_clock.cc:648] HybridClock initialized: now 1781992239495719 us; error 28 us; skew 500 ppm
I20260620 21:50:39.497867 5911 webserver.cc:460] Webserver started at http://127.0.193.254:43053/ using document root <none> and password file <none>
I20260620 21:50:39.498474 5911 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:39.498571 5911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:39.498813 5911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:39.500996 5911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "da41e4c92fae4896a3a3620ce1f40d2e"
format_stamp: "Formatted at 2026-06-20 21:50:39 on dist-test-slave-kk5c"
server_key: "e01a8bc4034d7b3aba8575c58417ad82"
server_key_iv: "c422bdee401871882df9bf83d130b7ff"
server_key_version: "encryptionkey@0"
I20260620 21:50:39.501603 5911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "da41e4c92fae4896a3a3620ce1f40d2e"
format_stamp: "Formatted at 2026-06-20 21:50:39 on dist-test-slave-kk5c"
server_key: "e01a8bc4034d7b3aba8575c58417ad82"
server_key_iv: "c422bdee401871882df9bf83d130b7ff"
server_key_version: "encryptionkey@0"
I20260620 21:50:39.505609 5911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:39.508225 5925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:39.509641 5911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:39.509795 5911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "da41e4c92fae4896a3a3620ce1f40d2e"
format_stamp: "Formatted at 2026-06-20 21:50:39 on dist-test-slave-kk5c"
server_key: "e01a8bc4034d7b3aba8575c58417ad82"
server_key_iv: "c422bdee401871882df9bf83d130b7ff"
server_key_version: "encryptionkey@0"
I20260620 21:50:39.509953 5911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:39.532122 5911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:39.532879 5911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:39.533100 5911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:39.541779 5911 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:39685
I20260620 21:50:39.541805 5977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:39685 every 8 connection(s)
I20260620 21:50:39.542791 5911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:39.546306 5978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:39.549854 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 5911
I20260620 21:50:39.550037 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:39.550320 775 external_mini_cluster.cc:1428] Setting key ca30a1ee2967511090af5fefae3d87a8
I20260620 21:50:39.552838 5978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e: Bootstrap starting.
I20260620 21:50:39.555099 5978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:39.555956 5978 log.cc:824] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:39.557950 5978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e: No bootstrap required, opened a new log
I20260620 21:50:39.560122 5981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:39.551804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.0.193.254:39685)
0620 21:50:39.552513 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:39.552528 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:39.552567 (+ 39us) server_negotiation.cc:407] Connection header received
0620 21:50:39.553351 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:39.553370 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:39.553789 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:39.554146 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:39.554586 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:39.555739 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:39.556719 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:39.557216 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:39.557396 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:39.558316 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:39.558321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:39.558335 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:39.558735 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:39.558805 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:39.558827 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:39.558945 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:39.559069 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 21:50:39.559374 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":141,"threads_started":1}
I20260620 21:50:39.561803 5978 raft_consensus.cc:348] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da41e4c92fae4896a3a3620ce1f40d2e" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39685 } }
I20260620 21:50:39.562076 5978 raft_consensus.cc:374] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:39.562134 5978 raft_consensus.cc:729] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da41e4c92fae4896a3a3620ce1f40d2e, State: Initialized, Role: FOLLOWER
I20260620 21:50:39.562611 5978 consensus_queue.cc:260] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da41e4c92fae4896a3a3620ce1f40d2e" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39685 } }
I20260620 21:50:39.562769 5978 raft_consensus.cc:388] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:39.562844 5978 raft_consensus.cc:482] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:39.562945 5978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:39.563235 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:50:39.564226 5978 raft_consensus.cc:504] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da41e4c92fae4896a3a3620ce1f40d2e" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39685 } }
I20260620 21:50:39.564661 5978 leader_election.cc:302] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da41e4c92fae4896a3a3620ce1f40d2e; no voters:
I20260620 21:50:39.565095 5978 leader_election.cc:288] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:39.565202 5983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:39.565513 5983 raft_consensus.cc:686] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [term 1 LEADER]: Becoming Leader. State: Replica: da41e4c92fae4896a3a3620ce1f40d2e, State: Running, Role: LEADER
I20260620 21:50:39.565865 5983 consensus_queue.cc:237] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da41e4c92fae4896a3a3620ce1f40d2e" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39685 } }
I20260620 21:50:39.566435 5978 sys_catalog.cc:564] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:39.567912 5985 sys_catalog.cc:455] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader da41e4c92fae4896a3a3620ce1f40d2e. Latest consensus state: current_term: 1 leader_uuid: "da41e4c92fae4896a3a3620ce1f40d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da41e4c92fae4896a3a3620ce1f40d2e" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39685 } } }
I20260620 21:50:39.567912 5984 sys_catalog.cc:455] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da41e4c92fae4896a3a3620ce1f40d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da41e4c92fae4896a3a3620ce1f40d2e" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39685 } } }
I20260620 21:50:39.568099 5984 sys_catalog.cc:458] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:39.568099 5985 sys_catalog.cc:458] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:39.568545 5995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:39.571913 5995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:39.578097 5995 catalog_manager.cc:1347] Generated new cluster ID: 6e4d11424ffd422390dc9e63f8cdeceb
I20260620 21:50:39.578190 5995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:39.615525 5995 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:39.617007 5995 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:39.625126 5995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da41e4c92fae4896a3a3620ce1f40d2e: Generated new TSK 0
I20260620 21:50:39.625934 5995 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 21:50:39.657353 6008 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 21:50:39.657491 6008 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 21:50:39.658113 5981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:39.639624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.0.193.254:39685)
0620 21:50:39.639766 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:39.639772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:39.639931 (+ 159us) server_negotiation.cc:407] Connection header received
0620 21:50:39.640162 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:39.640166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:39.640228 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:39.640346 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:50:39.640649 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:39.641513 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:39.642680 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:39.643084 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:39.643227 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:39.643716 (+ 489us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:50:39.657667 (+ 13951us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41805/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 21:50:39.657968 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40252: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41805/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 21:50:39.658208 5981 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40252: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41805/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 21:50:39.659473 6007 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:39.639514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39685 (local address 127.0.0.1:40252)
0620 21:50:39.639829 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:39.639844 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:39.639992 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:39.640420 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:39.640422 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:39.640429 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:50:39.640537 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:39.640542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:39.641846 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:39.641849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:39.642541 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:39.642550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:39.643356 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:39.643360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:39.643440 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:39.643494 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:50:39.657919 (+ 14425us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41805/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 21:50:39.658145 (+ 226us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.193.254:39685: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41805/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":250,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.193.254:39685: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41805/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 21:50:39.660151 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 5911
2026-06-20T21:50:39Z chronyd exiting
I20260620 21:50:39.697168 775 test_util.cc:175] -----------------------------------------------
I20260620 21:50:39.697261 775 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781992210835692-775-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (357 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 21:50:39.698287 775 mini_oidc.cc:152] Starting JWKS server
I20260620 21:50:39.699174 775 webserver.cc:460] Webserver started at https://127.0.0.1:40237/ using document root <none> and password file <none>
I20260620 21:50:39.699329 775 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 21:50:39.699646 775 webserver.cc:460] Webserver started at http://127.0.0.1:46399/ using document root <none> and password file <none>
2026-06-20T21:50:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:39Z Disabled control of system clock
I20260620 21:50:39.708877 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36159
--webserver_interface=127.0.193.254
--webserver_port=0
--builtin_ntp_servers=127.0.193.212:36263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:36159
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40237/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/testchainca.cert with env {}
W20260620 21:50:39.826115 6014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:39.826431 6014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:39.826476 6014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:39.830754 6014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:39.830822 6014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:39.830837 6014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:39.830854 6014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:39.830869 6014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:50:39.830885 6014 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:50:39.830899 6014 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40237/jwks.json
I20260620 21:50:39.836463 6014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:36159
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40237/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:39.837481 6014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:39.838608 6014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:39.847752 6020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:39.847757 6019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:39.847882 6014 server_base.cc:1034] running on GCE node
W20260620 21:50:39.847757 6022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:39.848692 6014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:39.849900 6014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:39.851095 6014 hybrid_clock.cc:648] HybridClock initialized: now 1781992239851064 us; error 46 us; skew 500 ppm
I20260620 21:50:39.857191 6014 webserver.cc:460] Webserver started at http://127.0.193.254:40277/ using document root <none> and password file <none>
I20260620 21:50:39.857873 6014 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:39.857944 6014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:39.858165 6014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:39.860389 6014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "ef9d53878e6842558c5cb2a2554cfa16"
format_stamp: "Formatted at 2026-06-20 21:50:39 on dist-test-slave-kk5c"
server_key: "30b1857c2c7ff618587b1c0b6f8b5a2b"
server_key_iv: "c2c4456f3a9322c01a7d40eb32674cd4"
server_key_version: "encryptionkey@0"
I20260620 21:50:39.860968 6014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "ef9d53878e6842558c5cb2a2554cfa16"
format_stamp: "Formatted at 2026-06-20 21:50:39 on dist-test-slave-kk5c"
server_key: "30b1857c2c7ff618587b1c0b6f8b5a2b"
server_key_iv: "c2c4456f3a9322c01a7d40eb32674cd4"
server_key_version: "encryptionkey@0"
I20260620 21:50:39.864773 6014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:39.867401 6028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:39.868729 6014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:39.868981 6014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "ef9d53878e6842558c5cb2a2554cfa16"
format_stamp: "Formatted at 2026-06-20 21:50:39 on dist-test-slave-kk5c"
server_key: "30b1857c2c7ff618587b1c0b6f8b5a2b"
server_key_iv: "c2c4456f3a9322c01a7d40eb32674cd4"
server_key_version: "encryptionkey@0"
I20260620 21:50:39.869098 6014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:39.909500 6014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:39.910251 6014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:39.910471 6014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:39.918121 6014 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:36159
I20260620 21:50:39.918176 6080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:36159 every 8 connection(s)
I20260620 21:50:39.919133 6014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:39.922490 6081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:39.925806 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 6014
I20260620 21:50:39.925974 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:39.926265 775 external_mini_cluster.cc:1428] Setting key 1a9baf560655dc327251362145a17001
I20260620 21:50:39.928826 6081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16: Bootstrap starting.
I20260620 21:50:39.931160 6081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:39.931993 6081 log.cc:824] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:39.933965 6081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16: No bootstrap required, opened a new log
I20260620 21:50:39.936349 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:39.927723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.0.193.254:36159)
0620 21:50:39.928443 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:39.928461 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:39.928498 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:50:39.929208 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:39.929296 (+ 88us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:39.929805 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:39.930267 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:39.930730 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:39.931874 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:39.932824 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:39.933305 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:39.933489 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:39.934475 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:39.934481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:39.934495 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:39.934893 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:39.934952 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:39.934973 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:39.935054 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:39.935285 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 21:50:39.935623 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":140,"threads_started":1}
I20260620 21:50:39.938208 6081 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9d53878e6842558c5cb2a2554cfa16" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36159 } }
I20260620 21:50:39.938470 6081 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:39.938546 6081 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9d53878e6842558c5cb2a2554cfa16, State: Initialized, Role: FOLLOWER
I20260620 21:50:39.939051 6081 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9d53878e6842558c5cb2a2554cfa16" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36159 } }
I20260620 21:50:39.939257 6081 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:39.939337 6081 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:39.939441 6081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:39.939947 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:50:39.940732 6081 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9d53878e6842558c5cb2a2554cfa16" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36159 } }
I20260620 21:50:39.941172 6081 leader_election.cc:302] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef9d53878e6842558c5cb2a2554cfa16; no voters:
I20260620 21:50:39.941592 6081 leader_election.cc:288] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:39.941679 6086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:39.941893 6086 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [term 1 LEADER]: Becoming Leader. State: Replica: ef9d53878e6842558c5cb2a2554cfa16, State: Running, Role: LEADER
I20260620 21:50:39.942219 6086 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9d53878e6842558c5cb2a2554cfa16" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36159 } }
I20260620 21:50:39.942970 6081 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:39.943845 6087 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef9d53878e6842558c5cb2a2554cfa16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9d53878e6842558c5cb2a2554cfa16" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36159 } } }
I20260620 21:50:39.944149 6088 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9d53878e6842558c5cb2a2554cfa16. Latest consensus state: current_term: 1 leader_uuid: "ef9d53878e6842558c5cb2a2554cfa16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9d53878e6842558c5cb2a2554cfa16" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36159 } } }
I20260620 21:50:39.944274 6088 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:39.944342 6087 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:39.944696 6095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:39.947477 6095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:39.952824 6095 catalog_manager.cc:1347] Generated new cluster ID: a8f1c7d8a557413fb28254d96e50eb12
I20260620 21:50:39.952900 6095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:39.964939 6095 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:39.966233 6095 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:39.977525 6095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef9d53878e6842558c5cb2a2554cfa16: Generated new TSK 0
I20260620 21:50:39.978376 6095 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 21:50:40.005720 6111 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 21:50:40.005861 6111 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 21:50:40.006614 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:39.988857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.0.193.254:36159)
0620 21:50:39.988994 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:39.989000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:39.989199 (+ 199us) server_negotiation.cc:407] Connection header received
0620 21:50:39.989335 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:39.989338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:39.989393 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:39.989510 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:50:39.989725 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:39.990562 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:39.991595 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:39.991947 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:39.992048 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:39.992355 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:50:40.006104 (+ 13749us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 21:50:40.006471 (+ 367us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59168: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 21:50:40.006685 6084 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59168: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 21:50:40.006814 6110 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:39.988771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36159 (local address 127.0.0.1:59168)
0620 21:50:39.989114 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:39.989126 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:39.989222 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:39.989516 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:39.989518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:39.989524 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:50:39.989623 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:39.989628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:39.990690 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:39.990693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:39.991471 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:39.991480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:39.992065 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:39.992068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:39.992151 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:39.992175 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:50:40.006433 (+ 14258us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 21:50:40.006599 (+ 166us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.193.254:36159: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40237/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20260620 21:50:40.007918 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 6014
2026-06-20T21:50:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (349 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 21:50:40.047617 775 mini_oidc.cc:152] Starting JWKS server
I20260620 21:50:40.048504 775 webserver.cc:460] Webserver started at https://127.0.0.1:43691/ using document root <none> and password file <none>
I20260620 21:50:40.048614 775 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 21:50:40.048870 775 webserver.cc:460] Webserver started at http://127.0.0.1:33939/ using document root <none> and password file <none>
2026-06-20T21:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:40Z Disabled control of system clock
I20260620 21:50:40.058234 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:39839
--webserver_interface=127.0.193.254
--webserver_port=0
--builtin_ntp_servers=127.0.193.212:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:39839
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43691/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 21:50:40.175638 6117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:40.175949 6117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:40.175990 6117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:40.180119 6117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:40.180190 6117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:40.180207 6117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:40.180224 6117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:40.180239 6117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:50:40.180256 6117 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:50:40.180270 6117 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43691/jwks.json
I20260620 21:50:40.185379 6117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:39839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:39839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43691/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:40.186493 6117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:40.188017 6117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:40.197510 6123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:40.197530 6125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:40.197530 6122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:40.197721 6117 server_base.cc:1034] running on GCE node
I20260620 21:50:40.198478 6117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:40.199745 6117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:40.200947 6117 hybrid_clock.cc:648] HybridClock initialized: now 1781992240200935 us; error 32 us; skew 500 ppm
I20260620 21:50:40.203151 6117 webserver.cc:460] Webserver started at http://127.0.193.254:34317/ using document root <none> and password file <none>
I20260620 21:50:40.203815 6117 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:40.203914 6117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:40.204154 6117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:40.206349 6117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "b3cbdc43ef294ac0a056bd57ee616ec0"
format_stamp: "Formatted at 2026-06-20 21:50:40 on dist-test-slave-kk5c"
server_key: "92b339eed43b100816805694e5a992b5"
server_key_iv: "2a55bf52be026dcf83b2cddfb8fcab15"
server_key_version: "encryptionkey@0"
I20260620 21:50:40.206983 6117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "b3cbdc43ef294ac0a056bd57ee616ec0"
format_stamp: "Formatted at 2026-06-20 21:50:40 on dist-test-slave-kk5c"
server_key: "92b339eed43b100816805694e5a992b5"
server_key_iv: "2a55bf52be026dcf83b2cddfb8fcab15"
server_key_version: "encryptionkey@0"
I20260620 21:50:40.211061 6117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 21:50:40.213806 6131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:40.215478 6117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:40.215629 6117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "b3cbdc43ef294ac0a056bd57ee616ec0"
format_stamp: "Formatted at 2026-06-20 21:50:40 on dist-test-slave-kk5c"
server_key: "92b339eed43b100816805694e5a992b5"
server_key_iv: "2a55bf52be026dcf83b2cddfb8fcab15"
server_key_version: "encryptionkey@0"
I20260620 21:50:40.215759 6117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:40.242794 6117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:40.243602 6117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:40.243817 6117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:40.251770 6117 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:39839
I20260620 21:50:40.251775 6183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:39839 every 8 connection(s)
I20260620 21:50:40.252902 6117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:40.255553 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 6117
I20260620 21:50:40.255676 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:40.255988 775 external_mini_cluster.cc:1428] Setting key b89913c4fe113a223caa7cbecf83b89f
I20260620 21:50:40.256734 6184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:40.263949 6184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0: Bootstrap starting.
I20260620 21:50:40.265955 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:40.257432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38014 (local address 127.0.193.254:39839)
0620 21:50:40.258098 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:40.258114 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:40.258151 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:50:40.258828 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:40.258846 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:40.259365 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:40.259673 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:40.260144 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:40.261146 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:40.262389 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:40.262872 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:40.262992 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:40.263957 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:40.263963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:40.263977 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:40.264512 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:40.264568 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:40.264589 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:40.264674 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:40.264849 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 21:50:40.265161 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":160,"threads_started":1}
I20260620 21:50:40.266407 6184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:40.267153 6184 log.cc:824] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:40.268950 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:50:40.269179 6184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0: No bootstrap required, opened a new log
I20260620 21:50:40.272735 6184 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cbdc43ef294ac0a056bd57ee616ec0" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39839 } }
I20260620 21:50:40.273001 6184 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:40.273085 6184 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cbdc43ef294ac0a056bd57ee616ec0, State: Initialized, Role: FOLLOWER
I20260620 21:50:40.273564 6184 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cbdc43ef294ac0a056bd57ee616ec0" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39839 } }
I20260620 21:50:40.273721 6184 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:40.273815 6184 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:40.273917 6184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:40.274935 6184 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cbdc43ef294ac0a056bd57ee616ec0" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39839 } }
I20260620 21:50:40.275368 6184 leader_election.cc:302] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3cbdc43ef294ac0a056bd57ee616ec0; no voters:
I20260620 21:50:40.275707 6184 leader_election.cc:288] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:40.276022 6189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:40.276214 6189 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [term 1 LEADER]: Becoming Leader. State: Replica: b3cbdc43ef294ac0a056bd57ee616ec0, State: Running, Role: LEADER
I20260620 21:50:40.276526 6189 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cbdc43ef294ac0a056bd57ee616ec0" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39839 } }
I20260620 21:50:40.277127 6184 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:40.278390 6191 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3cbdc43ef294ac0a056bd57ee616ec0. Latest consensus state: current_term: 1 leader_uuid: "b3cbdc43ef294ac0a056bd57ee616ec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cbdc43ef294ac0a056bd57ee616ec0" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39839 } } }
I20260620 21:50:40.278416 6190 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3cbdc43ef294ac0a056bd57ee616ec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cbdc43ef294ac0a056bd57ee616ec0" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39839 } } }
I20260620 21:50:40.278522 6191 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:40.278528 6190 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:40.278965 6198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:40.281774 6198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:40.287757 6198 catalog_manager.cc:1347] Generated new cluster ID: 57a39161399c4d83bc783601a5017777
I20260620 21:50:40.287843 6198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:40.295642 6198 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:40.297008 6198 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:40.301105 6198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3cbdc43ef294ac0a056bd57ee616ec0: Generated new TSK 0
I20260620 21:50:40.301856 6198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:50:40.321126 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:40.315661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.0.193.254:39839)
0620 21:50:40.315811 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:40.315816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:40.316029 (+ 213us) server_negotiation.cc:407] Connection header received
0620 21:50:40.316193 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:40.316195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:40.316260 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:40.316372 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:50:40.316688 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:40.317237 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:40.318518 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:40.318742 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:40.318862 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:40.319259 (+ 397us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:50:40.320701 (+ 1442us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43691/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 21:50:40.320990 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38022: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43691/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 21:50:40.321235 6187 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38022: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43691/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 21:50:40.323975 6213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:40.315573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39839 (local address 127.0.0.1:38022)
0620 21:50:40.315909 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:40.315925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:40.316057 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:40.316388 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:40.316392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:40.316400 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:50:40.316537 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:40.316543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:40.317371 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:40.317375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:40.318387 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:40.318398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:40.318873 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:40.318877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:40.318976 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:40.319005 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:50:40.320899 (+ 1894us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43691/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 21:50:40.323816 (+ 2917us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.193.254:39839: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43691/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":2650,"thread_start_us":137,"threads_started":1}
I20260620 21:50:40.324415 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 6117
E20260620 21:50:40.335085 6214 webserver.cc:533] Webserver: sslize failed:
E20260620 21:50:40.335173 6214 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T21:50:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (317 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T21:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:40Z Disabled control of system clock
I20260620 21:50:40.373587 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:37431
--webserver_interface=127.0.193.254
--webserver_port=0
--builtin_ntp_servers=127.0.193.212:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:37431
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/testchainca.cert with env {}
W20260620 21:50:40.492897 6218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:40.493228 6218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:40.493279 6218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:40.497984 6218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:40.498071 6218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:40.498088 6218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:40.498106 6218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:40.498122 6218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 21:50:40.498139 6218 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 21:50:40.498193 6218 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 21:50:40.504114 6218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:37431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:37431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:40.505328 6218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:40.506467 6218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:40.516062 6226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:40.516062 6224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:40.516081 6223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:40.516125 6218 server_base.cc:1034] running on GCE node
I20260620 21:50:40.516961 6218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:40.518093 6218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:40.519371 6218 hybrid_clock.cc:648] HybridClock initialized: now 1781992240519327 us; error 51 us; skew 500 ppm
I20260620 21:50:40.521467 6218 webserver.cc:460] Webserver started at http://127.0.193.254:33873/ using document root <none> and password file <none>
I20260620 21:50:40.522071 6218 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:40.522135 6218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:40.522310 6218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:40.524497 6218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "95e391e3a0ea4ae0ac8a391bea313f11"
format_stamp: "Formatted at 2026-06-20 21:50:40 on dist-test-slave-kk5c"
server_key: "69cd419249a6631458c6435ee41125cd"
server_key_iv: "5677d3f653f66c52e1436d28e1fb3217"
server_key_version: "encryptionkey@0"
I20260620 21:50:40.525071 6218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "95e391e3a0ea4ae0ac8a391bea313f11"
format_stamp: "Formatted at 2026-06-20 21:50:40 on dist-test-slave-kk5c"
server_key: "69cd419249a6631458c6435ee41125cd"
server_key_iv: "5677d3f653f66c52e1436d28e1fb3217"
server_key_version: "encryptionkey@0"
I20260620 21:50:40.529178 6218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 21:50:40.532117 6232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:40.533613 6218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:40.533778 6218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "95e391e3a0ea4ae0ac8a391bea313f11"
format_stamp: "Formatted at 2026-06-20 21:50:40 on dist-test-slave-kk5c"
server_key: "69cd419249a6631458c6435ee41125cd"
server_key_iv: "5677d3f653f66c52e1436d28e1fb3217"
server_key_version: "encryptionkey@0"
I20260620 21:50:40.533922 6218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:40.589018 6218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:40.589793 6218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:40.590008 6218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:40.597882 6218 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:37431
I20260620 21:50:40.597898 6284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:37431 every 8 connection(s)
I20260620 21:50:40.598905 6218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:40.600700 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 6218
I20260620 21:50:40.600844 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:40.601138 775 external_mini_cluster.cc:1428] Setting key 43e76bb8638c493e72ec6974ce3b0fe7
I20260620 21:50:40.602342 6285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:40.608181 6285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11: Bootstrap starting.
I20260620 21:50:40.610740 6285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:40.610852 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:40.602484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.0.193.254:37431)
0620 21:50:40.603155 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:40.603176 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:40.603247 (+ 71us) server_negotiation.cc:407] Connection header received
0620 21:50:40.603914 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:40.603939 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:40.604320 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:40.604667 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:40.605095 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:40.606341 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:40.607415 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:40.607904 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:40.608079 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:40.609033 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:40.609038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:40.609051 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:50:40.609478 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:40.609538 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:50:40.609560 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:40.609681 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:40.609831 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 21:50:40.610111 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":135,"threads_started":1}
I20260620 21:50:40.611655 6285 log.cc:824] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:40.614068 6285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11: No bootstrap required, opened a new log
I20260620 21:50:40.614224 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:50:40.617836 6285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e391e3a0ea4ae0ac8a391bea313f11" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 37431 } }
I20260620 21:50:40.618130 6285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:40.618221 6285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e391e3a0ea4ae0ac8a391bea313f11, State: Initialized, Role: FOLLOWER
I20260620 21:50:40.618736 6285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e391e3a0ea4ae0ac8a391bea313f11" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 37431 } }
I20260620 21:50:40.619092 6285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:40.619318 6285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:40.619470 6285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:40.620538 6285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e391e3a0ea4ae0ac8a391bea313f11" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 37431 } }
I20260620 21:50:40.620936 6285 leader_election.cc:302] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95e391e3a0ea4ae0ac8a391bea313f11; no voters:
I20260620 21:50:40.621323 6285 leader_election.cc:288] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:40.621488 6290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:40.621760 6290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [term 1 LEADER]: Becoming Leader. State: Replica: 95e391e3a0ea4ae0ac8a391bea313f11, State: Running, Role: LEADER
I20260620 21:50:40.622140 6290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e391e3a0ea4ae0ac8a391bea313f11" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 37431 } }
I20260620 21:50:40.622756 6285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:40.624145 6291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95e391e3a0ea4ae0ac8a391bea313f11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e391e3a0ea4ae0ac8a391bea313f11" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 37431 } } }
I20260620 21:50:40.624315 6291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:40.624809 6299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:40.625268 6292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95e391e3a0ea4ae0ac8a391bea313f11. Latest consensus state: current_term: 1 leader_uuid: "95e391e3a0ea4ae0ac8a391bea313f11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95e391e3a0ea4ae0ac8a391bea313f11" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 37431 } } }
I20260620 21:50:40.625541 6292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:40.628386 6299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:40.634804 6299 catalog_manager.cc:1347] Generated new cluster ID: 052717cb4af648859fe2a993814aafa4
I20260620 21:50:40.634895 6299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:40.648212 6299 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:40.649559 6299 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:40.661234 6299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95e391e3a0ea4ae0ac8a391bea313f11: Generated new TSK 0
I20260620 21:50:40.662081 6299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:50:40.792315 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:40.674598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.0.193.254:37431)
0620 21:50:40.674746 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:40.674753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:40.674930 (+ 177us) server_negotiation.cc:407] Connection header received
0620 21:50:40.675106 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:40.675109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:40.675157 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:40.675311 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0620 21:50:40.675561 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:40.676099 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:40.677367 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:40.677606 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:40.677695 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:40.678055 (+ 360us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 21:50:40.791885 (+113830us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 21:50:40.792182 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40102: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 21:50:40.792526 6288 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40102: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 21:50:40.792635 6314 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:40.674502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:37431 (local address 127.0.0.1:40102)
0620 21:50:40.674824 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:40.674840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:40.674967 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:40.675321 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:40.675324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:40.675333 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 21:50:40.675464 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:40.675471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:40.676215 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:40.676219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:40.677220 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:40.677232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:40.677737 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:40.677748 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:40.677851 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:40.677882 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 21:50:40.792147 (+114265us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 21:50:40.792396 (+ 249us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.193.254:37431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":120,"threads_started":1}
I20260620 21:50:40.793484 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 6218
2026-06-20T21:50:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (448 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (140 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (133 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:50:41 dist-test-slave-kk5c krb5kdc[6326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:50:41 dist-test-slave-kk5c krb5kdc[6326](info): set up 2 sockets
Jun 20 21:50:41 dist-test-slave-kk5c krb5kdc[6326](info): commencing operation
krb5kdc: starting...
W20260620 21:50:42.175616 775 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:50:42 dist-test-slave-kk5c krb5kdc[6326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:50:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:50:42.306876 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:38297
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:38297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:42.422449 6339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:42.422782 6339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:42.422825 6339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:42.427083 6339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:42.427158 6339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:42.427179 6339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:42.427263 6339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:42.427292 6339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:42.432850 6339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:38297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:38297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:42.434020 6339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:42.435101 6339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:42.443802 6344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:42.443802 6347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:42.443840 6345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:42.444061 6339 server_base.cc:1034] running on GCE node
I20260620 21:50:42.444775 6339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:42.445995 6339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:42.447232 6339 hybrid_clock.cc:648] HybridClock initialized: now 1781992242447180 us; error 26 us; skew 500 ppm
Jun 20 21:50:42 dist-test-slave-kk5c krb5kdc[6326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:42.450116 6339 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:50:42.451066 6339 webserver.cc:460] Webserver started at http://127.0.193.254:36875/ using document root <none> and password file <none>
I20260620 21:50:42.451700 6339 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:42.451794 6339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:42.452028 6339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:42.454102 6339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "dd3ea7254cd14400a81c9ec799ca87f6"
format_stamp: "Formatted at 2026-06-20 21:50:42 on dist-test-slave-kk5c"
server_key: "47dd7bafd5a83477a4409974c21d474c"
server_key_iv: "249ad63aef6556f292f92d6c1789978a"
server_key_version: "encryptionkey@0"
I20260620 21:50:42.454686 6339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "dd3ea7254cd14400a81c9ec799ca87f6"
format_stamp: "Formatted at 2026-06-20 21:50:42 on dist-test-slave-kk5c"
server_key: "47dd7bafd5a83477a4409974c21d474c"
server_key_iv: "249ad63aef6556f292f92d6c1789978a"
server_key_version: "encryptionkey@0"
I20260620 21:50:42.458685 6339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 21:50:42.461414 6354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:42.462702 6339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:50:42.462944 6339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "dd3ea7254cd14400a81c9ec799ca87f6"
format_stamp: "Formatted at 2026-06-20 21:50:42 on dist-test-slave-kk5c"
server_key: "47dd7bafd5a83477a4409974c21d474c"
server_key_iv: "249ad63aef6556f292f92d6c1789978a"
server_key_version: "encryptionkey@0"
I20260620 21:50:42.463070 6339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:42.483232 6339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:42.486651 6339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:42.486876 6339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:42.494349 6339 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:38297
I20260620 21:50:42.494665 6406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:38297 every 8 connection(s)
I20260620 21:50:42.495438 6339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:42.498302 6407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:42.504086 6407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6: Bootstrap starting.
I20260620 21:50:42.504330 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 6339
I20260620 21:50:42.504468 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:42.504755 775 external_mini_cluster.cc:1428] Setting key 6df75185ff821e5d8e6ab35ee8376d66
I20260620 21:50:42.506857 6407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:42.507663 6407 log.cc:824] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:42.510047 6407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6: No bootstrap required, opened a new log
Jun 20 21:50:42 dist-test-slave-kk5c krb5kdc[6326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:42.514523 6407 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3ea7254cd14400a81c9ec799ca87f6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38297 } }
I20260620 21:50:42.514827 6407 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:42.514936 6407 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3ea7254cd14400a81c9ec799ca87f6, State: Initialized, Role: FOLLOWER
I20260620 21:50:42.515522 6407 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3ea7254cd14400a81c9ec799ca87f6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38297 } }
I20260620 21:50:42.515713 6407 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:42.515786 6407 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:42.515895 6407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:42.516918 6407 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3ea7254cd14400a81c9ec799ca87f6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38297 } }
I20260620 21:50:42.517303 6407 leader_election.cc:302] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd3ea7254cd14400a81c9ec799ca87f6; no voters:
I20260620 21:50:42.517730 6407 leader_election.cc:288] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:42.517840 6412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:42.518069 6412 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [term 1 LEADER]: Becoming Leader. State: Replica: dd3ea7254cd14400a81c9ec799ca87f6, State: Running, Role: LEADER
I20260620 21:50:42.518453 6412 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3ea7254cd14400a81c9ec799ca87f6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38297 } }
I20260620 21:50:42.519281 6407 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:42.520387 6414 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd3ea7254cd14400a81c9ec799ca87f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3ea7254cd14400a81c9ec799ca87f6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38297 } } }
I20260620 21:50:42.520797 6414 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:42.520709 6413 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd3ea7254cd14400a81c9ec799ca87f6. Latest consensus state: current_term: 1 leader_uuid: "dd3ea7254cd14400a81c9ec799ca87f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3ea7254cd14400a81c9ec799ca87f6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38297 } } }
I20260620 21:50:42.521111 6413 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:42.521226 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:42.506213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.0.193.254:38297)
0620 21:50:42.506713 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:42.506731 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:42.506768 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:50:42.507583 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:42.507603 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:42.508022 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:42.508342 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:42.509095 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:42.510618 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:42.511621 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:42.512406 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:42.512575 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:42.515362 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:42.515368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:42.515385 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:42.515419 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:42.517419 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:42.518012 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:42.518021 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:42.518031 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:42.518199 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:42.518581 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:42.518584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:42.518586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:42.519146 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:42.519381 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:42.519678 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 21:50:42.520015 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":166,"threads_started":1}
I20260620 21:50:42.521688 6421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:42.524706 6421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:42.530923 6421 catalog_manager.cc:1347] Generated new cluster ID: 2c409ad93c024a1c9199062d22e140ad
I20260620 21:50:42.531035 6421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:42.540907 6421 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:42.542318 6421 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:42.558357 6421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd3ea7254cd14400a81c9ec799ca87f6: Generated new TSK 0
I20260620 21:50:42.559273 6421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:50:42.594861 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38297
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:42.738257 6433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:42.738570 6433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:42.738621 6433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:42.743484 6433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:42.743563 6433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:42.743670 6433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:42.749998 6433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:42.751277 6433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:42.752399 6433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:42.763234 6438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:42.763273 6433 server_base.cc:1034] running on GCE node
W20260620 21:50:42.763283 6439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:42.763435 6441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:42.764024 6433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:42.764678 6433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:42.765913 6433 hybrid_clock.cc:648] HybridClock initialized: now 1781992242765850 us; error 48 us; skew 500 ppm
Jun 20 21:50:42 dist-test-slave-kk5c krb5kdc[6326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:42.769256 6433 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:50:42.770279 6433 webserver.cc:460] Webserver started at http://127.0.193.193:43913/ using document root <none> and password file <none>
I20260620 21:50:42.770865 6433 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:42.770970 6433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:42.771257 6433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:42.773445 6433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "f3b3fd5a7ef84428afbf4ba8d224f797"
format_stamp: "Formatted at 2026-06-20 21:50:42 on dist-test-slave-kk5c"
server_key: "77bcb20b9f1dd1d9467e9f8ff3a083be"
server_key_iv: "a4f8a71643d82336d329995ec0487105"
server_key_version: "encryptionkey@0"
I20260620 21:50:42.774037 6433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "f3b3fd5a7ef84428afbf4ba8d224f797"
format_stamp: "Formatted at 2026-06-20 21:50:42 on dist-test-slave-kk5c"
server_key: "77bcb20b9f1dd1d9467e9f8ff3a083be"
server_key_iv: "a4f8a71643d82336d329995ec0487105"
server_key_version: "encryptionkey@0"
I20260620 21:50:42.778007 6433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:50:42.780580 6448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:42.781939 6433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:50:42.782115 6433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "f3b3fd5a7ef84428afbf4ba8d224f797"
format_stamp: "Formatted at 2026-06-20 21:50:42 on dist-test-slave-kk5c"
server_key: "77bcb20b9f1dd1d9467e9f8ff3a083be"
server_key_iv: "a4f8a71643d82336d329995ec0487105"
server_key_version: "encryptionkey@0"
I20260620 21:50:42.782245 6433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:42.796509 6433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:42.800474 6433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:42.800719 6433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:42.801378 6433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:42.802286 6433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:42.802364 6433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:42.802441 6433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:42.802486 6433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:42.813176 6433 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:33699
I20260620 21:50:42.813337 6561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:33699 every 8 connection(s)
I20260620 21:50:42.814158 6433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:42.823614 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 6433
I20260620 21:50:42.823812 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:50:42.824116 775 external_mini_cluster.cc:1428] Setting key 5d969821b537fbf36c54b5a5d98aa994
Jun 20 21:50:42 dist-test-slave-kk5c krb5kdc[6326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:42.830068 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:42.816218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:34133 (local address 127.0.193.254:38297)
0620 21:50:42.816386 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:42.816390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:42.817256 (+ 866us) server_negotiation.cc:407] Connection header received
0620 21:50:42.818858 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:42.818862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:42.818924 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:42.818994 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:42.820447 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:42.821004 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:42.822223 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:42.822458 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:42.822577 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:42.825805 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:42.825809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:42.825812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:42.825837 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:42.827483 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:42.828015 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:42.828020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:42.828022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:42.828118 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:42.828611 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:42.828616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:42.828617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:42.828984 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:42.829096 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:42.829676 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 21:50:42.829835 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 21:50:42.830981 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:42.816574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:38297 (local address 127.0.193.193:34133)
0620 21:50:42.817037 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:42.817080 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:42.818604 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:42.819176 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:42.819235 (+ 59us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:42.819634 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:42.820288 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:42.820304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:42.821148 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:42.821168 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:42.822051 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:42.822060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:42.822622 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:42.822626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:42.822807 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:42.823551 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:42.823570 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:42.825621 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:42.827614 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:42.827619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:42.827632 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:42.827912 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:42.828340 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:42.828343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:42.828344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:42.828501 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:42.829120 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:42.829126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:42.829423 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 21:50:42.829602 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 21:50:42.829909 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":116,"threads_started":1}
I20260620 21:50:42.832394 6562 heartbeater.cc:347] Connected to a master server at 127.0.193.254:38297
I20260620 21:50:42.832690 6562 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:42.833333 6562 heartbeater.cc:511] Master 127.0.193.254:38297 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
I20260620 21:50:42.835127 6371 ts_manager.cc:194] Registered new tserver with Master: f3b3fd5a7ef84428afbf4ba8d224f797 (127.0.193.193:33699)
I20260620 21:50:42.836479 6371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:34133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:50:42.854754 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38297
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:42.973024 6567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:42.973356 6567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:42.973443 6567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:42.977783 6567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:42.977890 6567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:42.977989 6567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:42.983635 6567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:42.984872 6567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:42.986043 6567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:42.995831 6575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:42.995855 6573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:42.995859 6572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:42.996061 6567 server_base.cc:1034] running on GCE node
I20260620 21:50:42.996610 6567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:42.997381 6567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:42.998598 6567 hybrid_clock.cc:648] HybridClock initialized: now 1781992242998577 us; error 35 us; skew 500 ppm
Jun 20 21:50:42 dist-test-slave-kk5c krb5kdc[6326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:43.001693 6567 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:50:43.002666 6567 webserver.cc:460] Webserver started at http://127.0.193.194:45141/ using document root <none> and password file <none>
I20260620 21:50:43.003409 6567 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:43.003508 6567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:43.003763 6567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:43.006121 6567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "f09b57d30dbc4bbe8bef827f91482784"
format_stamp: "Formatted at 2026-06-20 21:50:42 on dist-test-slave-kk5c"
server_key: "fa10ddfb41d6b0ae8ab2a9f771866c1c"
server_key_iv: "279bdee546938a610756e38ffefcad31"
server_key_version: "encryptionkey@0"
I20260620 21:50:43.006745 6567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "f09b57d30dbc4bbe8bef827f91482784"
format_stamp: "Formatted at 2026-06-20 21:50:42 on dist-test-slave-kk5c"
server_key: "fa10ddfb41d6b0ae8ab2a9f771866c1c"
server_key_iv: "279bdee546938a610756e38ffefcad31"
server_key_version: "encryptionkey@0"
I20260620 21:50:43.010763 6567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:50:43.013518 6582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:43.014876 6567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:50:43.015031 6567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "f09b57d30dbc4bbe8bef827f91482784"
format_stamp: "Formatted at 2026-06-20 21:50:42 on dist-test-slave-kk5c"
server_key: "fa10ddfb41d6b0ae8ab2a9f771866c1c"
server_key_iv: "279bdee546938a610756e38ffefcad31"
server_key_version: "encryptionkey@0"
I20260620 21:50:43.015151 6567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:43.033654 6567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:43.036949 6567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:43.037112 6567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:43.037663 6567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:43.038533 6567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:43.038656 6567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:43.038759 6567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:43.038810 6567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:43.047789 6567 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:40999
I20260620 21:50:43.047896 6695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:40999 every 8 connection(s)
I20260620 21:50:43.048758 6567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:43.051894 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 6567
I20260620 21:50:43.052018 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:50:43.052325 775 external_mini_cluster.cc:1428] Setting key d03af7d16bfc9a84a09883dd5bac4636
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
I20260620 21:50:43.065933 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:43.050737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:36727 (local address 127.0.193.254:38297)
0620 21:50:43.050883 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:43.050886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:43.051731 (+ 845us) server_negotiation.cc:407] Connection header received
0620 21:50:43.053566 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:43.053569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:43.053615 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:43.053711 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:43.055486 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:43.056353 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:43.057731 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:43.058188 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:43.058322 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:43.061512 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:43.061517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:43.061521 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:43.061558 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:43.063249 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:43.063891 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:43.063893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:43.063895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:43.063977 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:43.064451 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:43.064456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:43.064458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:43.064683 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:43.064774 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:43.065623 (+ 849us) server_negotiation.cc:299] Negotiation successful
0620 21:50:43.065744 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:50:43.067155 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:43.051043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:38297 (local address 127.0.193.194:36727)
0620 21:50:43.051550 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:43.051625 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:43.053345 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:43.053922 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:43.053934 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:43.054530 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:43.055309 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:43.055334 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:43.056503 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:43.056506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:43.057571 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:43.057586 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:43.058437 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:43.058442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:43.058641 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:43.059403 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:43.059421 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:43.061309 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:43.063395 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:43.063401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:43.063416 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:43.063757 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:43.064069 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:43.064072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:43.064073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:43.064270 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:43.064815 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:43.064821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:43.065203 (+ 382us) client_negotiation.cc:769] Sending connection context
0620 21:50:43.065548 (+ 345us) client_negotiation.cc:240] Negotiation successful
0620 21:50:43.065965 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":181,"threads_started":1}
I20260620 21:50:43.068425 6696 heartbeater.cc:347] Connected to a master server at 127.0.193.254:38297
I20260620 21:50:43.068701 6696 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:43.069247 6696 heartbeater.cc:511] Master 127.0.193.254:38297 requested a full tablet report, sending...
I20260620 21:50:43.070370 6371 ts_manager.cc:194] Registered new tserver with Master: f09b57d30dbc4bbe8bef827f91482784 (127.0.193.194:40999)
I20260620 21:50:43.071341 6371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:36727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:50:43.083283 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38297
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:43.197124 6701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:43.197461 6701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:43.197539 6701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:43.201714 6701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:43.201820 6701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:43.201923 6701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:43.207918 6701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:43.209231 6701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:43.210304 6701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:43.220309 6706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:43.220319 6707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:43.220492 6709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:43.220949 6701 server_base.cc:1034] running on GCE node
I20260620 21:50:43.221644 6701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:43.222517 6701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:43.223752 6701 hybrid_clock.cc:648] HybridClock initialized: now 1781992243223714 us; error 52 us; skew 500 ppm
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:43.226645 6701 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:50:43.227638 6701 webserver.cc:460] Webserver started at http://127.0.193.195:44925/ using document root <none> and password file <none>
I20260620 21:50:43.228159 6701 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:43.228216 6701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:43.228389 6701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:43.230391 6701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "0765f0c16f8f4df6a8cfffeb9c0ada30"
format_stamp: "Formatted at 2026-06-20 21:50:43 on dist-test-slave-kk5c"
server_key: "8d78f4dc6d7d5a2fa6e180e2ba9a71ac"
server_key_iv: "44f8356ccac2ba1029dcef7a845e094c"
server_key_version: "encryptionkey@0"
I20260620 21:50:43.230906 6701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "0765f0c16f8f4df6a8cfffeb9c0ada30"
format_stamp: "Formatted at 2026-06-20 21:50:43 on dist-test-slave-kk5c"
server_key: "8d78f4dc6d7d5a2fa6e180e2ba9a71ac"
server_key_iv: "44f8356ccac2ba1029dcef7a845e094c"
server_key_version: "encryptionkey@0"
I20260620 21:50:43.234814 6701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:50:43.237403 6716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:43.238875 6701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:50:43.238982 6701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "0765f0c16f8f4df6a8cfffeb9c0ada30"
format_stamp: "Formatted at 2026-06-20 21:50:43 on dist-test-slave-kk5c"
server_key: "8d78f4dc6d7d5a2fa6e180e2ba9a71ac"
server_key_iv: "44f8356ccac2ba1029dcef7a845e094c"
server_key_version: "encryptionkey@0"
I20260620 21:50:43.239080 6701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:43.251362 6701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:43.254279 6701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:43.254436 6701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:43.254957 6701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:43.255826 6701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:43.255873 6701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:43.255915 6701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:43.255937 6701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:43.265198 6701 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:37079
I20260620 21:50:43.265211 6829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:37079 every 8 connection(s)
I20260620 21:50:43.266149 6701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:50:43.269870 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 6701
I20260620 21:50:43.270008 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:50:43.270289 775 external_mini_cluster.cc:1428] Setting key a752def6475770058ccbaac890b05b86
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:43.281374 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:43.268337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:46701 (local address 127.0.193.254:38297)
0620 21:50:43.268483 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:43.268487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:43.269425 (+ 938us) server_negotiation.cc:407] Connection header received
0620 21:50:43.271027 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:43.271030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:43.271087 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:43.271226 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:43.272805 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:43.273371 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:43.274564 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:43.274801 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:43.274904 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:43.277574 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:43.277577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:43.277580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:43.277604 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:43.279123 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:43.279676 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:43.279678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:43.279679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:43.279755 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:43.280205 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:43.280208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:43.280209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:43.280402 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:43.280490 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:43.281058 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 21:50:43.281244 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:50:43.282244 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:43.268742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:38297 (local address 127.0.193.195:46701)
0620 21:50:43.269240 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:43.269283 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:43.270731 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:43.271484 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:43.271495 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:43.272004 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:43.272613 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:43.272633 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:43.273494 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:43.273496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:43.274442 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:43.274452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:43.274957 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:43.274960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:43.275107 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:43.275742 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:43.275758 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:43.277379 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:43.279300 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:43.279305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:43.279319 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:43.279579 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:43.279834 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:43.279836 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:43.279838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:43.279968 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:43.280523 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:43.280528 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:43.280805 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 21:50:43.280979 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 21:50:43.281260 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":121,"threads_started":1}
I20260620 21:50:43.283691 6830 heartbeater.cc:347] Connected to a master server at 127.0.193.254:38297
I20260620 21:50:43.283953 6830 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:43.284629 6830 heartbeater.cc:511] Master 127.0.193.254:38297 requested a full tablet report, sending...
I20260620 21:50:43.285756 6371 ts_manager.cc:194] Registered new tserver with Master: 0765f0c16f8f4df6a8cfffeb9c0ada30 (127.0.193.195:37079)
I20260620 21:50:43.286284 6371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:46701
I20260620 21:50:43.297178 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/test-admin.keytab.
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:43.316246 775 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 21:50:43.319723 6837 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:43.317885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57300 (local address 127.0.193.193:33699)
0620 21:50:43.318230 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:43.318245 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:43.318269 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:50:43.318319 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:43.318324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:43.318583 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:43.318745 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:43.319438 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57300: BlockingRecv error: recv got EOF from 127.0.0.1:57300 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":141,"threads_started":1}
W20260620 21:50:43.319911 775 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/krb5cc' of type FILE
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:43.321388 775 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:43.322518 775 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.193@KRBTEST.COM
I20260620 21:50:43.334386 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:43.323633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.0.193.193:33699)
0620 21:50:43.323789 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:43.323793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:43.323923 (+ 130us) server_negotiation.cc:407] Connection header received
0620 21:50:43.324084 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:43.324087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:43.324137 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:43.324226 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:43.324818 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:43.325618 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:43.326493 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:43.326896 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:43.327016 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:43.329911 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:43.329923 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:43.329939 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:43.329970 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:43.331962 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:43.332558 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:43.332564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:43.332566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:43.332661 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:43.333041 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:43.333044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:43.333045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:43.333325 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:43.333537 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:43.334025 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 21:50:43.334171 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 21:50:43.340104 6837 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:43.338376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.0.193.193:33699)
0620 21:50:43.338506 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:43.338510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:43.338732 (+ 222us) server_negotiation.cc:407] Connection header received
0620 21:50:43.338791 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:43.338793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:43.338917 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:43.339012 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:43.339821 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57324: BlockingRecv error: recv got EOF from 127.0.0.1:57324 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 21:50:43.340512 775 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/krb5cc)
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:43.341924 775 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:43.343341 775 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.193@KRBTEST.COM
I20260620 21:50:43.354295 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:43.345189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57334 (local address 127.0.193.193:33699)
0620 21:50:43.345342 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:43.345347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:43.345363 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:50:43.345578 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:43.345588 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:43.345630 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:43.345721 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:43.346511 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:43.347056 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:43.347944 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:43.348183 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:43.348302 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:43.350987 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:43.350991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:43.350992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:43.351021 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:43.352491 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:43.353043 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:43.353053 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:43.353055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:43.353132 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:43.353423 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:43.353427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:43.353429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:43.353611 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:43.353760 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:43.353989 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 21:50:43.354109 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 21:50:43.357553 6837 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:43.356371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57350 (local address 127.0.193.193:33699)
0620 21:50:43.356491 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:43.356494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:43.356510 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:50:43.356639 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:43.356641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:43.356676 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:43.356754 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:43.357413 (+ 659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57350: BlockingRecv error: recv got EOF from 127.0.0.1:57350 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 21:50:43.357668 775 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781992210835692-775-0/krb5kdc/krb5cc' of type FILE
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:43.359159 775 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:43.360528 775 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.193@KRBTEST.COM
I20260620 21:50:43.371435 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:43.361635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.0.193.193:33699)
0620 21:50:43.361782 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:43.361786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:43.361904 (+ 118us) server_negotiation.cc:407] Connection header received
0620 21:50:43.362082 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:43.362092 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:43.362135 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:43.362227 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:43.362781 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:43.363393 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:43.364414 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:43.364650 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:43.364790 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:43.367851 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:43.367854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:43.367857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:43.367887 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:43.369602 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:43.370103 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:43.370107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:43.370109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:43.370223 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:43.370515 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:43.370518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:43.370520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:43.370707 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:43.370863 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:43.371104 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 21:50:43.371273 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:50:43.372509 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 6433
I20260620 21:50:43.378369 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 6567
I20260620 21:50:43.384527 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 6701
I20260620 21:50:43.389804 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 6339
2026-06-20T21:50:43Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2324 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6850](info): set up 2 sockets
Jun 20 21:50:43 dist-test-slave-kk5c krb5kdc[6850](info): commencing operation
krb5kdc: starting...
W20260620 21:50:44.475684 775 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:50:44 dist-test-slave-kk5c krb5kdc[6850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.254.keytab.
Entry for principal oryx/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.254.keytab.
I20260620 21:50:44.607474 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:44219
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.254.keytab
--principal=oryx/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:44219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:44.722960 6863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:44.723323 6863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:44.723395 6863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:44.727716 6863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:44.727793 6863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:44.727813 6863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:44.727831 6863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:44.727846 6863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:44.734261 6863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:44219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.254.keytab
--principal=oryx/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:44.735522 6863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:44.736610 6863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:44.746143 6869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:44.745971 6868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:44.746006 6871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:44.746181 6863 server_base.cc:1034] running on GCE node
I20260620 21:50:44.746965 6863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:44.748322 6863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:44.749557 6863 hybrid_clock.cc:648] HybridClock initialized: now 1781992244749539 us; error 39 us; skew 500 ppm
Jun 20 21:50:44 dist-test-slave-kk5c krb5kdc[6850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992244, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:44.752979 6863 init.cc:377] Logged in from keytab as oryx/127.0.193.254@KRBTEST.COM (short username oryx)
I20260620 21:50:44.754053 6863 webserver.cc:460] Webserver started at http://127.0.193.254:36503/ using document root <none> and password file <none>
I20260620 21:50:44.754700 6863 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:44.754812 6863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:44.755069 6863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:44.757369 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "c1af966f8a6c4dfb80f698a80ca810ef"
format_stamp: "Formatted at 2026-06-20 21:50:44 on dist-test-slave-kk5c"
server_key: "a62b3af9228d42a16b03c8d6a3b4c1be"
server_key_iv: "c3ce4a08da3b6bebd0245ff12d56e669"
server_key_version: "encryptionkey@0"
I20260620 21:50:44.757999 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "c1af966f8a6c4dfb80f698a80ca810ef"
format_stamp: "Formatted at 2026-06-20 21:50:44 on dist-test-slave-kk5c"
server_key: "a62b3af9228d42a16b03c8d6a3b4c1be"
server_key_iv: "c3ce4a08da3b6bebd0245ff12d56e669"
server_key_version: "encryptionkey@0"
I20260620 21:50:44.762430 6863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:50:44.765331 6878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:44.766654 6863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 21:50:44.766861 6863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "c1af966f8a6c4dfb80f698a80ca810ef"
format_stamp: "Formatted at 2026-06-20 21:50:44 on dist-test-slave-kk5c"
server_key: "a62b3af9228d42a16b03c8d6a3b4c1be"
server_key_iv: "c3ce4a08da3b6bebd0245ff12d56e669"
server_key_version: "encryptionkey@0"
I20260620 21:50:44.766993 6863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:44.783433 6863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:44.786863 6863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:44.787088 6863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:44.794341 6863 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:44219
I20260620 21:50:44.794660 6930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:44219 every 8 connection(s)
I20260620 21:50:44.795405 6863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:44.798272 6931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:44.805142 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 6863
I20260620 21:50:44.805297 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:44.805491 6931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef: Bootstrap starting.
I20260620 21:50:44.805621 775 external_mini_cluster.cc:1428] Setting key 8c0110d308a7688b4129e2fc899eeb94
I20260620 21:50:44.808279 6931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:44.809046 6931 log.cc:824] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:44.811064 6931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef: No bootstrap required, opened a new log
Jun 20 21:50:44 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
I20260620 21:50:44.815096 6931 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } }
I20260620 21:50:44.815466 6931 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:44.815527 6931 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1af966f8a6c4dfb80f698a80ca810ef, State: Initialized, Role: FOLLOWER
I20260620 21:50:44.815987 6931 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } }
I20260620 21:50:44.816120 6931 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:44.816206 6931 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:44.816301 6931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:44.817533 6931 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } }
I20260620 21:50:44.817914 6931 leader_election.cc:302] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1af966f8a6c4dfb80f698a80ca810ef; no voters:
I20260620 21:50:44.818244 6931 leader_election.cc:288] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:44.818372 6936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:44.818692 6936 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [term 1 LEADER]: Becoming Leader. State: Replica: c1af966f8a6c4dfb80f698a80ca810ef, State: Running, Role: LEADER
I20260620 21:50:44.819144 6936 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } }
I20260620 21:50:44.819756 6931 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:44.820935 6936 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1af966f8a6c4dfb80f698a80ca810ef. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } }
I20260620 21:50:44.821203 6936 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:44.821828 6945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:44.823132 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:44.807228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.0.193.254:44219)
0620 21:50:44.807702 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:44.807718 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:44.807755 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:50:44.808542 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:44.808562 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:44.808914 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:44.809272 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:44.810198 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:44.811271 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:44.812170 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:44.812654 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:44.812838 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:44.816106 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:44.816115 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:44.816138 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:44.816201 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:44.818316 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:44.818871 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:44.818878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:44.818885 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:44.819049 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:44.819446 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:44.819449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:44.819451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:44.820240 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:44.820505 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:44.821437 (+ 932us) server_negotiation.cc:299] Negotiation successful
0620 21:50:44.821863 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":147,"threads_started":1}
I20260620 21:50:44.823939 6937 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } }
I20260620 21:50:44.824395 6937 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:44.826022 6945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:44.832362 6945 catalog_manager.cc:1347] Generated new cluster ID: 30b65a13dabc4460a8959069c8331c8c
I20260620 21:50:44.832454 6945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:44.851914 6945 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:44.852936 6945 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:44.858006 6945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef: Generated new TSK 0
I20260620 21:50:44.858732 6945 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.193.keytab.
Entry for principal oryx/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.193.keytab.
I20260620 21:50:44.900584 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:44219
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.193.keytab
--principal=oryx/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:45.034786 6957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:45.035075 6957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:45.035122 6957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:45.039356 6957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:45.039430 6957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:45.039492 6957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:45.045076 6957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.193.keytab
--principal=oryx/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:44219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:45.046263 6957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:45.047366 6957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:45.057389 6965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:45.057307 6962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:45.057425 6957 server_base.cc:1034] running on GCE node
W20260620 21:50:45.057307 6963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:45.057978 6957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:45.058636 6957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:45.059849 6957 hybrid_clock.cc:648] HybridClock initialized: now 1781992245059807 us; error 52 us; skew 500 ppm
Jun 20 21:50:45 dist-test-slave-kk5c krb5kdc[6850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992245, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:45.066432 6957 init.cc:377] Logged in from keytab as oryx/127.0.193.193@KRBTEST.COM (short username oryx)
I20260620 21:50:45.067515 6957 webserver.cc:460] Webserver started at http://127.0.193.193:43735/ using document root <none> and password file <none>
I20260620 21:50:45.068178 6957 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:45.068250 6957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:45.068486 6957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:45.070937 6957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "0df4a56890b34a72be8f690aa2009dcf"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "950ebce2e008181e58b522ef144bd0ba"
server_key_iv: "7cf2c72e19a4b648b3c14ee2f3d3ecd0"
server_key_version: "encryptionkey@0"
I20260620 21:50:45.071671 6957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "0df4a56890b34a72be8f690aa2009dcf"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "950ebce2e008181e58b522ef144bd0ba"
server_key_iv: "7cf2c72e19a4b648b3c14ee2f3d3ecd0"
server_key_version: "encryptionkey@0"
I20260620 21:50:45.076146 6957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:50:45.078985 6972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:45.080624 6957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:45.080811 6957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "0df4a56890b34a72be8f690aa2009dcf"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "950ebce2e008181e58b522ef144bd0ba"
server_key_iv: "7cf2c72e19a4b648b3c14ee2f3d3ecd0"
server_key_version: "encryptionkey@0"
I20260620 21:50:45.080957 6957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:45.094820 6957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:45.098410 6957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:45.098637 6957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:45.099298 6957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:45.100159 6957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:45.100232 6957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:45.100303 6957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:45.100353 6957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:45.110366 6957 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:41503
I20260620 21:50:45.110509 7085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:41503 every 8 connection(s)
I20260620 21:50:45.111562 6957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:45.118831 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 6957
I20260620 21:50:45.118994 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:50:45.119393 775 external_mini_cluster.cc:1428] Setting key bf2496c8ca223234729f08c53e61fa90
Jun 20 21:50:45 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992245, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
I20260620 21:50:45.127872 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.113748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:43559 (local address 127.0.193.254:44219)
0620 21:50:45.113931 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:45.113936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:45.114832 (+ 896us) server_negotiation.cc:407] Connection header received
0620 21:50:45.116215 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:45.116218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:45.116268 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:45.116352 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:45.117623 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.118165 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.119660 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.119961 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.120055 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.123551 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:45.123554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:45.123557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:45.123584 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:45.125322 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.126065 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.126067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.126069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.126148 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.126578 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.126580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.126582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.126783 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:45.126858 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:45.127608 (+ 750us) server_negotiation.cc:299] Negotiation successful
0620 21:50:45.127722 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 21:50:45.129163 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.114101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.193:43559)
0620 21:50:45.114650 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:45.114703 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:45.116003 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:45.116526 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:45.116533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:45.116918 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:45.117474 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.117488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.118334 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.118337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.119520 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.119533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.120166 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.120170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.120448 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.121242 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:45.121260 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:45.123321 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:45.125467 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.125475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.125495 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.125941 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.126243 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.126249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.126254 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.126474 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.126892 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:45.126903 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:45.127312 (+ 409us) client_negotiation.cc:769] Sending connection context
0620 21:50:45.127545 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 21:50:45.127961 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for oryx/127.0.193.194@KRBTEST.COM; defaulting to no policy
I20260620 21:50:45.130771 7086 heartbeater.cc:347] Connected to a master server at 127.0.193.254:44219
I20260620 21:50:45.131094 7086 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:45.131672 7086 heartbeater.cc:511] Master 127.0.193.254:44219 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.193.194@KRBTEST.COM" created.
I20260620 21:50:45.133513 6894 ts_manager.cc:194] Registered new tserver with Master: 0df4a56890b34a72be8f690aa2009dcf (127.0.193.193:41503)
I20260620 21:50:45.135375 6894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:43559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.194.keytab.
Entry for principal oryx/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.194.keytab.
I20260620 21:50:45.152768 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:44219
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.194.keytab
--principal=oryx/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:45.267560 7091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:45.267889 7091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:45.267987 7091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:45.272215 7091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:45.272310 7091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:45.272397 7091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:45.277810 7091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.194.keytab
--principal=oryx/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:44219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:45.278963 7091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:45.280068 7091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:45.290251 7097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:45.290180 7099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:45.290558 7096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:45.290637 7091 server_base.cc:1034] running on GCE node
I20260620 21:50:45.291067 7091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:45.291844 7091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:45.293026 7091 hybrid_clock.cc:648] HybridClock initialized: now 1781992245293013 us; error 41 us; skew 500 ppm
Jun 20 21:50:45 dist-test-slave-kk5c krb5kdc[6850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992245, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:45.296145 7091 init.cc:377] Logged in from keytab as oryx/127.0.193.194@KRBTEST.COM (short username oryx)
I20260620 21:50:45.297237 7091 webserver.cc:460] Webserver started at http://127.0.193.194:32825/ using document root <none> and password file <none>
I20260620 21:50:45.298056 7091 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:45.298158 7091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:45.298393 7091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:45.300534 7091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "3738bd4c059c46f1985c0584596a312f"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "ffb78a1468948f442dcef564175c5486"
server_key_iv: "10a39007261ec22cde8a992d8ded420b"
server_key_version: "encryptionkey@0"
I20260620 21:50:45.301131 7091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "3738bd4c059c46f1985c0584596a312f"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "ffb78a1468948f442dcef564175c5486"
server_key_iv: "10a39007261ec22cde8a992d8ded420b"
server_key_version: "encryptionkey@0"
I20260620 21:50:45.305217 7091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:45.308136 7106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:45.309870 7091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:45.309995 7091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "3738bd4c059c46f1985c0584596a312f"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "ffb78a1468948f442dcef564175c5486"
server_key_iv: "10a39007261ec22cde8a992d8ded420b"
server_key_version: "encryptionkey@0"
I20260620 21:50:45.310096 7091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:45.321405 7091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:45.325279 7091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:45.325512 7091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:45.326139 7091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:45.327004 7091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:45.327075 7091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:45.327147 7091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:45.327229 7091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 21:50:45.336221 7091 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:39791
I20260620 21:50:45.336261 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:39791 every 8 connection(s)
I20260620 21:50:45.337224 7091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:45.339872 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 7091
I20260620 21:50:45.339973 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:50:45.340297 775 external_mini_cluster.cc:1428] Setting key d59da03e42bea56e07e4df4e3d767eac
Jun 20 21:50:45 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992245, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.193.195@KRBTEST.COM" created.
I20260620 21:50:45.353801 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.339365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:52951 (local address 127.0.193.254:44219)
0620 21:50:45.339510 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:45.339514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:45.340549 (+ 1035us) server_negotiation.cc:407] Connection header received
0620 21:50:45.342175 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:45.342179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:45.342259 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:45.342351 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:45.343776 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.344431 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.345694 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.345932 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.346049 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.348988 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:45.348992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:45.348994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:45.349019 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:45.350937 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.351705 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.351713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.351715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.351798 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.352169 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.352180 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.352182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.352451 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:45.352657 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:45.353281 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 21:50:45.353558 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:50:45.354939 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.339741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.194:52951)
0620 21:50:45.340313 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:45.340420 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:45.341952 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:45.342616 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:45.342625 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:45.343014 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:45.343617 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.343633 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.344596 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.344600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.345557 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.345567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.346100 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.346104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.346298 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.347070 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:45.347088 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:45.348767 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:45.351077 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.351085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.351105 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.351573 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.351896 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.351901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.351903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.352062 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.352658 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:45.352668 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:45.353008 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 21:50:45.353213 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 21:50:45.353812 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":178,"threads_started":1}
I20260620 21:50:45.356484 7220 heartbeater.cc:347] Connected to a master server at 127.0.193.254:44219
I20260620 21:50:45.356775 7220 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:45.357327 7220 heartbeater.cc:511] Master 127.0.193.254:44219 requested a full tablet report, sending...
I20260620 21:50:45.358620 6894 ts_manager.cc:194] Registered new tserver with Master: 3738bd4c059c46f1985c0584596a312f (127.0.193.194:39791)
I20260620 21:50:45.359567 6894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:52951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.195.keytab.
Entry for principal oryx/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.195.keytab.
I20260620 21:50:45.375689 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:44219
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.195.keytab
--principal=oryx/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:45.491123 7225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:45.491503 7225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:45.491547 7225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:45.495783 7225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:45.495865 7225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:45.495929 7225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:45.501487 7225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.195.keytab
--principal=oryx/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:44219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:45.502640 7225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:45.503724 7225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:45.514808 7233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:45.514835 7231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:45.514828 7225 server_base.cc:1034] running on GCE node
W20260620 21:50:45.515337 7230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:45.515935 7225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:45.516690 7225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:45.517899 7225 hybrid_clock.cc:648] HybridClock initialized: now 1781992245517874 us; error 39 us; skew 500 ppm
Jun 20 21:50:45 dist-test-slave-kk5c krb5kdc[6850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992245, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:45.521195 7225 init.cc:377] Logged in from keytab as oryx/127.0.193.195@KRBTEST.COM (short username oryx)
I20260620 21:50:45.522290 7225 webserver.cc:460] Webserver started at http://127.0.193.195:45271/ using document root <none> and password file <none>
I20260620 21:50:45.522886 7225 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:45.522982 7225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:45.523284 7225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:45.525465 7225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "9fb7ce4170754c97b58212aa70323c9f"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "03e5bb1d045e95eaed4ea47ca6f8154f"
server_key_iv: "ef66eaeb749a9d20f9be3e0c31893bad"
server_key_version: "encryptionkey@0"
I20260620 21:50:45.526067 7225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "9fb7ce4170754c97b58212aa70323c9f"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "03e5bb1d045e95eaed4ea47ca6f8154f"
server_key_iv: "ef66eaeb749a9d20f9be3e0c31893bad"
server_key_version: "encryptionkey@0"
I20260620 21:50:45.530036 7225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:50:45.532974 7240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:45.534492 7225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:50:45.534654 7225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "9fb7ce4170754c97b58212aa70323c9f"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "03e5bb1d045e95eaed4ea47ca6f8154f"
server_key_iv: "ef66eaeb749a9d20f9be3e0c31893bad"
server_key_version: "encryptionkey@0"
I20260620 21:50:45.534794 7225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:45.546666 7225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:45.550134 7225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:45.550369 7225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:45.551014 7225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:45.551949 7225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:45.552024 7225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:45.552098 7225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:45.552148 7225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:45.561863 7225 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:41471
I20260620 21:50:45.561887 7353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:41471 every 8 connection(s)
I20260620 21:50:45.562898 7225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:50:45.573185 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 7225
I20260620 21:50:45.573308 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:50:45.573737 775 external_mini_cluster.cc:1428] Setting key 29cf91372e74bfc0c7648e568cd23f65
Jun 20 21:50:45 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992245, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
I20260620 21:50:45.579360 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.565265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:58857 (local address 127.0.193.254:44219)
0620 21:50:45.565437 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:45.565440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:45.566429 (+ 989us) server_negotiation.cc:407] Connection header received
0620 21:50:45.568177 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:45.568180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:45.568241 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:45.568328 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:45.569620 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.570251 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.571549 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.571774 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.571888 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.574964 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:45.574968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:45.574971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:45.574999 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:45.576799 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.577395 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.577398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.577399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.577484 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.577892 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.577895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.577896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.578105 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:45.578192 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:45.578944 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 21:50:45.579083 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 21:50:45.580101 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.565556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.195:58857)
0620 21:50:45.566141 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:45.566251 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:45.567911 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:45.568529 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:45.568537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:45.568937 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:45.569449 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.569463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.570387 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.570391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.571419 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.571429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.571936 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.571939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.572109 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.572752 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:45.572769 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:45.574742 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:45.576928 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.576934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.576949 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.577273 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.577578 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.577580 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.577582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.577775 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.578245 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:45.578251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:45.578667 (+ 416us) client_negotiation.cc:769] Sending connection context
0620 21:50:45.578863 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 21:50:45.579140 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":137,"threads_started":1}
I20260620 21:50:45.581377 7354 heartbeater.cc:347] Connected to a master server at 127.0.193.254:44219
I20260620 21:50:45.581632 7354 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:45.582141 7354 heartbeater.cc:511] Master 127.0.193.254:44219 requested a full tablet report, sending...
I20260620 21:50:45.583280 6894 ts_manager.cc:194] Registered new tserver with Master: 9fb7ce4170754c97b58212aa70323c9f (127.0.193.195:41471)
I20260620 21:50:45.583899 6894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:58857
I20260620 21:50:45.588506 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.193.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.193.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.193.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.253.keytab.
Entry for principal oryx/127.0.193.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.253.keytab.
I20260620 21:50:45.619503 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.253:38733
--webserver_interface=127.0.193.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.253.keytab
--principal=oryx/127.0.193.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:44219,127.0.193.253:38733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:45.740832 7359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:45.741156 7359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:45.741200 7359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:45.745323 7359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:45.745395 7359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:45.745416 7359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:45.745433 7359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:45.745450 7359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:45.751006 7359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:44219,127.0.193.253:38733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.253.keytab
--principal=oryx/127.0.193.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.253:38733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.193.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
W20260620 21:50:45.751291 7359 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.193.254:44219,127.0.193.253:38733'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 21:50:45.752213 7359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:45.753219 7359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:45.762485 7365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:45.762485 7367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:45.762480 7364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:45.762931 7359 server_base.cc:1034] running on GCE node
I20260620 21:50:45.763685 7359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:45.764901 7359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:45.766100 7359 hybrid_clock.cc:648] HybridClock initialized: now 1781992245766072 us; error 39 us; skew 500 ppm
Jun 20 21:50:45 dist-test-slave-kk5c krb5kdc[6850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992245, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:45.769140 7359 init.cc:377] Logged in from keytab as oryx/127.0.193.253@KRBTEST.COM (short username oryx)
I20260620 21:50:45.770125 7359 webserver.cc:460] Webserver started at http://127.0.193.253:39325/ using document root <none> and password file <none>
I20260620 21:50:45.770718 7359 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:45.770792 7359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:45.771018 7359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:45.773348 7359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/data/instance:
uuid: "a225ae546cb44aa8a305b263966c267f"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "47223a070c5f0a2dbcfb72fa25a3fba0"
server_key_iv: "731ef3ebcb6e888585c23cffdb671a35"
server_key_version: "encryptionkey@0"
I20260620 21:50:45.773918 7359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/wal/instance:
uuid: "a225ae546cb44aa8a305b263966c267f"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "47223a070c5f0a2dbcfb72fa25a3fba0"
server_key_iv: "731ef3ebcb6e888585c23cffdb671a35"
server_key_version: "encryptionkey@0"
I20260620 21:50:45.777973 7359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:50:45.780719 7374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:45.782150 7359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:50:45.782361 7359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/wal
uuid: "a225ae546cb44aa8a305b263966c267f"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "47223a070c5f0a2dbcfb72fa25a3fba0"
server_key_iv: "731ef3ebcb6e888585c23cffdb671a35"
server_key_version: "encryptionkey@0"
I20260620 21:50:45.782495 7359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:45.798025 7359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:45.801892 7359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:45.802129 7359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:45.810268 7359 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.253:38733
I20260620 21:50:45.810568 7426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.253:38733 every 8 connection(s)
I20260620 21:50:45.811470 7359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/data/info.pb
I20260620 21:50:45.815011 7427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:45.817376 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 7359
I20260620 21:50:45.819857 7427 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:50:45 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992245, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.253@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
Jun 20 21:50:45 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.193.253@KRBTEST.COM
I20260620 21:50:45.832758 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.822092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.0.193.254:44219)
0620 21:50:45.822268 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:45.822275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:45.822550 (+ 275us) server_negotiation.cc:407] Connection header received
0620 21:50:45.822675 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:45.822678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:45.822739 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:45.822835 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:45.823767 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.824432 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.825715 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.826032 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.826156 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.828905 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:45.828908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:45.828910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:45.828933 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:45.830546 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.831131 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.831135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.831137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.831273 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.831619 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.831622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.831624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.831855 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:45.831970 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:45.832472 (+ 502us) server_negotiation.cc:299] Negotiation successful
0620 21:50:45.832592 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:50:45.833602 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.822014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.0.1:52340)
0620 21:50:45.822407 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:45.822458 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:45.822575 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:45.822892 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:45.822924 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:45.823414 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:45.823609 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.823622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.824571 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.824573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.825588 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.825598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.826174 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.826177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.826248 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.826918 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:45.826934 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:45.828685 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:45.830662 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.830673 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.830684 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.831027 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.831365 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.831369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.831372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.831513 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.831966 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:45.831973 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:45.832254 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 21:50:45.832434 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 21:50:45.832755 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":43,"spinlock_wait_cycles":5376,"thread_start_us":149,"threads_started":1}
I20260620 21:50:45.833604 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.819167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.0.193.253:38733)
0620 21:50:45.819940 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:45.819966 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:45.820011 (+ 45us) server_negotiation.cc:407] Connection header received
0620 21:50:45.820835 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:45.820860 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:45.821328 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:45.821700 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:45.822866 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.824188 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.825193 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.825824 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.825933 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.829277 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:45.829283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:45.829295 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:45.829317 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:45.831074 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.831587 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.831601 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.831604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.831678 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.831988 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.831993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.831995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.832269 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:45.832500 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:45.832745 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 21:50:45.833183 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":511,"thread_start_us":163,"threads_started":1}
I20260620 21:50:45.835001 7427 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:50:45 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992245, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.253@KRBTEST.COM for oryx/127.0.193.253@KRBTEST.COM
I20260620 21:50:45.843504 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.835701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.0.193.253:38733)
0620 21:50:45.835813 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:45.835819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:45.835831 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:50:45.835861 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:45.835866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:45.835905 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:45.835978 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:45.836397 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.837218 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.837995 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.838266 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.838369 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.840526 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:45.840530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:45.840533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:45.840560 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:45.841862 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.842249 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.842256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.842258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.842330 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.842584 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.842590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.842592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.842779 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:45.842885 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:45.843153 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 21:50:45.843337 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":37}
I20260620 21:50:45.843505 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.835561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:38733 (local address 127.0.0.1:42692)
0620 21:50:45.835668 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:45.835683 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:45.835757 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:45.836022 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:45.836024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:45.836172 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:45.836282 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.836288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.837323 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.837325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.837890 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.837898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.838373 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.838377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.838472 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.838982 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:45.838993 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:45.840394 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:45.841972 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.841978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.841981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.842159 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.842426 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.842429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.842430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.842502 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.842874 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:45.842877 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:45.842966 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 21:50:45.843108 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 21:50:45.843286 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 21:50:45.846504 7427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: Bootstrap starting.
I20260620 21:50:45.848886 7427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:45.849645 7427 log.cc:824] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:45.851871 7427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: No bootstrap required, opened a new log
I20260620 21:50:45.855086 7427 raft_consensus.cc:348] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } }
I20260620 21:50:45.855363 7427 raft_consensus.cc:374] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:45.855438 7427 raft_consensus.cc:729] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a225ae546cb44aa8a305b263966c267f, State: Initialized, Role: FOLLOWER
I20260620 21:50:45.855911 7427 consensus_queue.cc:260] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } }
I20260620 21:50:45.856416 7433 sys_catalog.cc:455] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } } }
I20260620 21:50:45.856575 7433 sys_catalog.cc:458] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:50:45.857161 7427 sys_catalog.cc:564] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: configured and running, proceeding with master startup.
W20260620 21:50:45.862012 7444 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:50:45.862126 7444 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:50:45.864782 7359 master_runner.cc:428] Detected that this master a225ae546cb44aa8a305b263966c267f is joining an existing cluster
I20260620 21:50:45.864853 7359 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.193.253:38733
I20260620 21:50:45.877854 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.867304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:38733 (local address 127.0.0.1:42696)
0620 21:50:45.867614 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:45.867631 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:45.867747 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:45.868057 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:45.868063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:45.868379 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:45.868548 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.868556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.869720 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.869723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.870991 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.871002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.871864 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.871867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.871966 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.872515 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:45.872530 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:45.873020 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:45.875880 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.875885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.875887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.876110 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.876436 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.876439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.876441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.876525 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.877136 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:45.877140 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:45.877248 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 21:50:45.877388 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 21:50:45.877574 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":132,"thread_start_us":129,"threads_started":1}
I20260620 21:50:45.877867 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.867245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.0.193.253:38733)
0620 21:50:45.867363 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:45.867368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:45.867724 (+ 356us) server_negotiation.cc:407] Connection header received
0620 21:50:45.867877 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:45.867880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:45.867933 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:45.868025 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:45.868781 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.869576 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.871148 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.871618 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.871727 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.873177 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:45.873184 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:45.873191 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:45.873217 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:45.875712 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.876220 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.876227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.876230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.876326 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.876624 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.876631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.876633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.876978 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:45.877145 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:45.877429 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 21:50:45.877701 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":46}
I20260620 21:50:45.878363 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.867260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.0.193.254:44219)
0620 21:50:45.867370 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:45.867375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:45.867816 (+ 441us) server_negotiation.cc:407] Connection header received
0620 21:50:45.867927 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:45.867930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:45.867981 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:45.868078 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:45.868867 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.869517 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.871736 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.871991 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.872120 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.873895 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:45.873900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:45.873903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:45.873934 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:45.876372 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.876923 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.876927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.876929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.877019 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.877333 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.877340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.877344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.877596 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:45.877701 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:45.878077 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 21:50:45.878235 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:50:45.878841 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.867180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.0.1:52342)
0620 21:50:45.867745 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:45.867757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:45.867836 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:45.868273 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:45.868276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:45.868525 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:45.868626 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.868632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.869668 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.869675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.871609 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.871618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.872135 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.872139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.872537 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.873073 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:45.873124 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:45.873692 (+ 568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:45.876514 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.876517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.876520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.876798 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.877126 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.877129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.877131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.877219 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.877750 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:45.877754 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:45.877848 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 21:50:45.878529 (+ 681us) client_negotiation.cc:240] Negotiation successful
0620 21:50:45.878671 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":170,"thread_start_us":126,"threads_started":1}
Jun 20 21:50:45 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992244, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.254@KRBTEST.COM for oryx/127.0.193.253@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.193.252@KRBTEST.COM; defaulting to no policy
I20260620 21:50:45.897292 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.884148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:38733 (local address 127.0.0.1:42704)
0620 21:50:45.884650 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:45.884708 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:45.884873 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:45.885304 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:45.885310 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:45.885947 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:45.886138 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.886148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.887366 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.887370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.888404 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.888417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.889312 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.889316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.889424 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.890104 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:45.890118 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:45.892516 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:45.895007 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.895022 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.895030 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.895458 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.895825 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.895828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.895830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.895937 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.896497 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:45.896505 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:45.896651 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 21:50:45.896840 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 21:50:45.897083 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":175,"threads_started":1}
I20260620 21:50:45.897493 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.884185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.0.193.253:38733)
0620 21:50:45.884351 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:45.884356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:45.884801 (+ 445us) server_negotiation.cc:407] Connection header received
0620 21:50:45.885031 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:45.885035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:45.885096 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:45.885196 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:45.886291 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.887218 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.888558 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.889017 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.889166 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.892716 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:45.892724 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:45.892731 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:45.892784 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:45.894843 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.895589 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.895594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.895597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.895707 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.896055 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.896059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.896061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.896363 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:45.896488 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:45.896940 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 21:50:45.897082 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 21:50:45.899519 6894 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.193.253:38733: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a225ae546cb44aa8a305b263966c267f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.193.253"
port: 38733
}
attrs {
promote: true
}
}
dest_uuid: "c1af966f8a6c4dfb80f698a80ca810ef"
cas_config_opid_index: -1
I20260620 21:50:45.900290 6894 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: NON_VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.193.252@KRBTEST.COM" created.
I20260620 21:50:45.902860 7456 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a225ae546cb44aa8a305b263966c267f (127.0.193.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: NON_VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: true } } }
I20260620 21:50:45.903821 7399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:45.905009 7456 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.193.253:38733
I20260620 21:50:45.905297 7399 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 FOLLOWER]: Refusing update from remote peer c1af966f8a6c4dfb80f698a80ca810ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 21:50:45.905268 7456 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: NON_VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: true } } }
I20260620 21:50:45.905364 7456 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:45.906092 7359 master.cc:537] Master@127.0.193.253:38733 shutting down...
I20260620 21:50:45.906304 7456 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: NON_VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 21:50:45.908713 6880 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 21:50:45.910075 6880 consensus_peers.cc:487] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef -> Peer a225ae546cb44aa8a305b263966c267f (127.0.193.253:38733): Couldn't send request to peer a225ae546cb44aa8a305b263966c267f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 21:50:45.910780 7359 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:45.911103 7359 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:45.911242 7359 tablet_replica.cc:330] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: stopping tablet replica
I20260620 21:50:45.927906 7359 master.cc:552] Master@127.0.193.253:38733 shutdown complete.
I20260620 21:50:45.927985 7359 master_runner.cc:305] Clearing existing system tablet
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.193.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.252.keytab.
Entry for principal oryx/127.0.193.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.252.keytab.
I20260620 21:50:45.928256 7359 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:45.928318 7359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:45.929210 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.252:42751
--webserver_interface=127.0.193.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.252.keytab
--principal=oryx/127.0.193.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:44219,127.0.193.253:38733,127.0.193.252:42751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:50:45.929908 7359 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 21:50:45.931010 7462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:45.931279 7359 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:45.931447 7359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:45.931488 7359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/wal
uuid: "a225ae546cb44aa8a305b263966c267f"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "47223a070c5f0a2dbcfb72fa25a3fba0"
server_key_iv: "731ef3ebcb6e888585c23cffdb671a35"
server_key_version: "encryptionkey@0"
I20260620 21:50:45.934026 7359 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:45.937631 7359 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 21:50:45.937788 7359 log.cc:1196] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 21:50:45.938169 7359 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: Deleting consensus metadata
I20260620 21:50:45.938385 7359 master_runner.cc:315] Copying system tablet from 127.0.193.254:44219
I20260620 21:50:45.939992 7359 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.193.254:44219
I20260620 21:50:45.952277 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.940702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.0.1:52356)
0620 21:50:45.941268 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:45.941289 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:45.941439 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:45.941796 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:45.941800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:45.942069 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:45.942333 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.942345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.943769 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.943773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.944779 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:45.944793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.945540 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.945544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:45.945650 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.946310 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:45.946327 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:45.946797 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:45.949264 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.949273 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.949278 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.950078 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.950511 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:45.950516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:45.950518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:45.950638 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:45.951626 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:45.951630 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:45.951748 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 21:50:45.951872 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 21:50:45.952045 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":198,"threads_started":1}
I20260620 21:50:45.952280 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:45.940842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52356 (local address 127.0.193.254:44219)
0620 21:50:45.941025 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:45.941032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:45.941401 (+ 369us) server_negotiation.cc:407] Connection header received
0620 21:50:45.941559 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:45.941563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:45.941632 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:45.941751 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:45.942508 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.943583 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.944951 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:45.945366 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:45.945505 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:45.947002 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:45.947008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:45.947011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:45.947042 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:45.948936 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.950239 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.950244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.950246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.950385 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:45.951151 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:45.951156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:45.951158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:45.951499 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:45.951610 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:45.951955 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 21:50:45.952106 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 21:50:45.953272 6915 tablet_copy_service.cc:140] P c1af966f8a6c4dfb80f698a80ca810ef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a225ae546cb44aa8a305b263966c267f ({username='oryx', principal='oryx/127.0.193.253@KRBTEST.COM'} at 127.0.0.1:52356)
I20260620 21:50:45.953586 6915 tablet_copy_service.cc:161] P c1af966f8a6c4dfb80f698a80ca810ef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a225ae546cb44aa8a305b263966c267f at {username='oryx', principal='oryx/127.0.193.253@KRBTEST.COM'} at 127.0.0.1:52356: session id = a225ae546cb44aa8a305b263966c267f-00000000000000000000000000000000
I20260620 21:50:45.955772 6915 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 21:50:45.957753 7359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 21:50:45.960270 7359 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: tablet copy: Starting download of 0 data blocks...
I20260620 21:50:45.960481 7359 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: tablet copy: Starting download of 1 WAL segments...
I20260620 21:50:45.962469 7359 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 21:50:45.964586 6915 tablet_copy_service.cc:342] P c1af966f8a6c4dfb80f698a80ca810ef: Request end of tablet copy session a225ae546cb44aa8a305b263966c267f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.193.253@KRBTEST.COM'} at 127.0.0.1:52356
I20260620 21:50:45.964737 6915 tablet_copy_service.cc:433] P c1af966f8a6c4dfb80f698a80ca810ef: ending tablet copy session a225ae546cb44aa8a305b263966c267f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a225ae546cb44aa8a305b263966c267f
I20260620 21:50:45.972600 7359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:45.974588 7472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:45.974776 7469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:45.974927 7359 server_base.cc:1034] running on GCE node
W20260620 21:50:45.979427 7470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:45.980051 7359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:45.980492 7359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:45.981652 7359 hybrid_clock.cc:648] HybridClock initialized: now 1781992245981672 us; error 59 us; skew 500 ppm
Jun 20 21:50:45 dist-test-slave-kk5c krb5kdc[6850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992245, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:45.983512 7359 init.cc:377] Logged in from keytab as oryx/127.0.193.253@KRBTEST.COM (short username oryx)
I20260620 21:50:45.984098 7359 webserver.cc:460] Webserver started at http://127.0.193.253:41385/ using document root <none> and password file <none>
I20260620 21:50:45.984356 7359 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:45.984417 7359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:45.986083 7359 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:45.987130 7479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:45.987447 7359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:45.987555 7359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/wal
uuid: "a225ae546cb44aa8a305b263966c267f"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "47223a070c5f0a2dbcfb72fa25a3fba0"
server_key_iv: "731ef3ebcb6e888585c23cffdb671a35"
server_key_version: "encryptionkey@0"
I20260620 21:50:45.987668 7359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:45.997193 7359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:45.999718 7359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:46.006363 7359 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.253:38733
I20260620 21:50:46.006408 7531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.253:38733 every 8 connection(s)
I20260620 21:50:46.007057 7359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-1/data/info.pb
I20260620 21:50:46.008635 7532 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:50:46.009577 7532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: Bootstrap starting.
I20260620 21:50:46.019088 7532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:50:46.019707 7532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: Bootstrap complete.
I20260620 21:50:46.020359 7532 raft_consensus.cc:348] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: NON_VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: true } }
I20260620 21:50:46.020509 7532 raft_consensus.cc:729] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a225ae546cb44aa8a305b263966c267f, State: Initialized, Role: LEARNER
I20260620 21:50:46.020635 7532 consensus_queue.cc:260] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: NON_VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: true } }
I20260620 21:50:46.021189 7535 sys_catalog.cc:455] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: NON_VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: true } } }
I20260620 21:50:46.021343 7532 sys_catalog.cc:564] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:46.021380 7535 sys_catalog.cc:458] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: This master's current role is: LEARNER
I20260620 21:50:46.024727 7546 catalog_manager.cc:1259] Loaded cluster ID: 30b65a13dabc4460a8959069c8331c8c
I20260620 21:50:46.024833 7546 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: loading cluster ID for follower catalog manager: success
I20260620 21:50:46.026614 7546 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: acquiring CA information for follower catalog manager: success
I20260620 21:50:46.027967 7546 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 21:50:46.094563 7461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:46.094888 7461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:46.094934 7461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:46.099380 7461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:46.099454 7461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:46.099474 7461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:46.099491 7461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:46.099542 7461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:46.105396 7461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:44219,127.0.193.253:38733,127.0.193.252:42751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.252.keytab
--principal=oryx/127.0.193.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.252:42751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.193.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:46.106607 7461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:46.107807 7461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:46.118089 7554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:46.118525 7551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:46.118688 7552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:46.119578 7461 server_base.cc:1034] running on GCE node
I20260620 21:50:46.120653 7461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:46.122539 7461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:46.123725 7461 hybrid_clock.cc:648] HybridClock initialized: now 1781992246123712 us; error 34 us; skew 500 ppm
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:46.126967 7461 init.cc:377] Logged in from keytab as oryx/127.0.193.252@KRBTEST.COM (short username oryx)
I20260620 21:50:46.128261 7461 webserver.cc:460] Webserver started at http://127.0.193.252:40091/ using document root <none> and password file <none>
I20260620 21:50:46.128976 7461 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:46.129184 7461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:46.129478 7461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:46.132015 7461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/data/instance:
uuid: "ab0459286b274ed183b16dbb8b46235a"
format_stamp: "Formatted at 2026-06-20 21:50:46 on dist-test-slave-kk5c"
server_key: "c2b30ab230671c2c2b9dd6c63f7a695d"
server_key_iv: "dc1b1a848e8c0772c3f9723f23de4231"
server_key_version: "encryptionkey@0"
I20260620 21:50:46.132715 7461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/wal/instance:
uuid: "ab0459286b274ed183b16dbb8b46235a"
format_stamp: "Formatted at 2026-06-20 21:50:46 on dist-test-slave-kk5c"
server_key: "c2b30ab230671c2c2b9dd6c63f7a695d"
server_key_iv: "dc1b1a848e8c0772c3f9723f23de4231"
server_key_version: "encryptionkey@0"
I20260620 21:50:46.137374 7461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:46.138273 7086 heartbeater.cc:503] Master 127.0.193.254:44219 was elected leader, sending a full tablet report...
I20260620 21:50:46.141336 7561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:46.142920 7461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:50:46.143105 7461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/wal
uuid: "ab0459286b274ed183b16dbb8b46235a"
format_stamp: "Formatted at 2026-06-20 21:50:46 on dist-test-slave-kk5c"
server_key: "c2b30ab230671c2c2b9dd6c63f7a695d"
server_key_iv: "dc1b1a848e8c0772c3f9723f23de4231"
server_key_version: "encryptionkey@0"
I20260620 21:50:46.143312 7461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:46.161131 7461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:46.165352 7461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:46.165637 7461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:46.174718 7461 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.252:42751
I20260620 21:50:46.175263 7613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.252:42751 every 8 connection(s)
I20260620 21:50:46.175940 7461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/data/info.pb
I20260620 21:50:46.179229 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 7461
I20260620 21:50:46.179738 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:46.185200 7614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.193.252@KRBTEST.COM
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.252@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
I20260620 21:50:46.197722 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.181036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.0.193.252:42751)
0620 21:50:46.181526 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.181542 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.181580 (+ 38us) server_negotiation.cc:407] Connection header received
0620 21:50:46.182651 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.182675 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.183183 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.183687 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.184559 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.186040 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.187025 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.188112 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.188287 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.192118 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.192124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.192135 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.192166 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.194348 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.194906 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.194919 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.194925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.195034 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.195411 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.195416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.195418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.196020 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.196252 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.196588 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.196901 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":152,"threads_started":1}
I20260620 21:50:46.199003 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.187874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52364 (local address 127.0.193.254:44219)
0620 21:50:46.188031 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.188036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.188580 (+ 544us) server_negotiation.cc:407] Connection header received
0620 21:50:46.188737 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.188739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.188785 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.188854 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.189915 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.190742 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.192138 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.192385 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.192514 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.195072 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.195075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.195078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.195103 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.197158 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.197648 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.197652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.197654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.197741 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.198080 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.198084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.198085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.198306 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.198377 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.198793 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.198893 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:50:46.200117 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.187608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.0.1:52364)
0620 21:50:46.188412 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.188473 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.188614 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.188942 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.188947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.189470 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.189720 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.189730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.190914 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.190918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.191987 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.191999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.192513 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.192517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.192685 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.193328 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.193340 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.194896 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:46.197292 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.197301 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.197309 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.197535 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.197852 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.197855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.197857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.197960 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.198452 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:46.198458 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:46.198568 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.198707 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.199935 (+ 1228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":199,"threads_started":1}
I20260620 21:50:46.201448 7614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.252@KRBTEST.COM for oryx/127.0.193.253@KRBTEST.COM
I20260620 21:50:46.212644 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.202018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:38733 (local address 127.0.0.1:42720)
0620 21:50:46.202154 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.202171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.202274 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.202792 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.202795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.202992 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.203093 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.203098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.204084 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.204087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.205210 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.205222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.205785 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.205789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.205894 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.206451 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.206463 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.208184 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:46.210614 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.210617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.210619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.210843 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.211433 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.211436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.211437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.211531 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.212139 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:46.212142 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:46.212278 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.212378 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.212510 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 21:50:46.213649 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.202123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.0.193.253:38733)
0620 21:50:46.202460 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.202463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.202475 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:50:46.202511 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.202513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.202553 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.202779 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.203245 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.203949 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.205364 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.205614 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.205789 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.208367 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.208371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.208374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.208402 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.210426 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.211110 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.211113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.211114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.211301 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.211733 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.211760 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.211761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.212005 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.212106 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.212507 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.213490 (+ 983us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":100,"threads_started":1}
I20260620 21:50:46.214771 7614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.252@KRBTEST.COM for oryx/127.0.193.252@KRBTEST.COM
I20260620 21:50:46.226712 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.215615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.0.193.252:42751)
0620 21:50:46.215787 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.215790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.215901 (+ 111us) server_negotiation.cc:407] Connection header received
0620 21:50:46.215968 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.215970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.216010 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.216082 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.216700 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.217216 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.218577 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.218864 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.218966 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.222265 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.222274 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.222277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.222302 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.224310 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.224903 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.224909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.224910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.225074 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.225768 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.225772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.225773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.225988 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.226086 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.226471 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.226583 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 21:50:46.226768 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.215517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:42751 (local address 127.0.0.1:34104)
0620 21:50:46.215785 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.215804 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.215905 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.216228 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.216230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.216477 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.216562 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.216568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.217736 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.217739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.218454 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.218464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.219020 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.219022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.219091 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.219900 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.219912 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.221924 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:46.224452 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.224455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.224456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.224669 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.225467 (+ 798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.225470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.225472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.225585 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.226123 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:46.226126 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:46.226216 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.226323 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.226451 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 21:50:46.231590 7614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: Bootstrap starting.
I20260620 21:50:46.234270 7614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:46.235076 7614 log.cc:824] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:46.237718 7614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: No bootstrap required, opened a new log
I20260620 21:50:46.241640 7614 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } }
I20260620 21:50:46.241909 7614 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:46.241947 7614 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0459286b274ed183b16dbb8b46235a, State: Initialized, Role: FOLLOWER
I20260620 21:50:46.242364 7614 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } }
I20260620 21:50:46.242894 7620 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } } }
I20260620 21:50:46.243075 7620 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:50:46.243702 7614 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [sys.catalog]: configured and running, proceeding with master startup.
W20260620 21:50:46.250558 7631 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:50:46.250771 7631 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:50:46.251384 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 6957
I20260620 21:50:46.256742 7461 master_runner.cc:428] Detected that this master ab0459286b274ed183b16dbb8b46235a is joining an existing cluster
I20260620 21:50:46.256850 7461 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.193.252:42751
I20260620 21:50:46.259423 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:41503
--local_ip_for_outbound_sockets=127.0.193.193
--tserver_master_addrs=127.0.193.254:44219
--webserver_port=43735
--webserver_interface=127.0.193.193
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.193.keytab
--principal=oryx/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.193.254:44219,127.0.193.253:38733
--tserver_master_addrs=127.0.193.254:44219,127.0.193.253:38733,127.0.193.252:42751 with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:50:46.274102 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.260569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.0.193.252:42751)
0620 21:50:46.260720 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.260725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.261013 (+ 288us) server_negotiation.cc:407] Connection header received
0620 21:50:46.261221 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.261225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.261279 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.261330 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.262078 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.262867 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.264380 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.264874 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.264988 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.268074 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.268078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.268081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.268101 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.270712 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.271555 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.271560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.271562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.271855 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.272446 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.272450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.272451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.273015 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.273144 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.273660 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.273843 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 21:50:46.275027 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.260108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:42751 (local address 127.0.0.1:34118)
0620 21:50:46.260901 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.260938 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.261113 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.261485 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.261489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.261739 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.261940 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.261949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.262999 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.263004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.264229 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.264243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.265207 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.265212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.265417 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.266371 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.266393 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.267950 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:46.270840 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.270847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.270850 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.271170 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.272096 (+ 926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.272099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.272101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.272306 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.273258 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:46.273269 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:46.273440 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.274606 (+ 1166us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.274802 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"spinlock_wait_cycles":1792,"thread_start_us":131,"threads_started":1}
I20260620 21:50:46.275132 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.260179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.0.193.253:38733)
0620 21:50:46.260708 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.260714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.261712 (+ 998us) server_negotiation.cc:407] Connection header received
0620 21:50:46.261914 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.261917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.261966 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.262019 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.262992 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.263976 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.265429 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.265980 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.266074 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.268442 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.268446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.268449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.268479 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.271815 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.272792 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.272796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.272798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.272924 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.273331 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.273334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.273337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.273861 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.273984 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.274451 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.274754 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405}
I20260620 21:50:46.276170 7640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.260337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:38733 (local address 127.0.0.1:42726)
0620 21:50:46.261625 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.261639 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.261798 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.262161 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.262164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.262500 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.262701 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.262712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.264348 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.264352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.265262 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.265272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.266246 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.266250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.266362 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.267141 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.267155 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.267720 (+ 565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:46.272100 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.272105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.272108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.272632 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.273060 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.273064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.273066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.273197 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.274079 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:46.274082 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:46.274246 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.275667 (+ 1421us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.275975 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1222,"thread_start_us":92,"threads_started":1}
I20260620 21:50:46.292289 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.261318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.0.1:52370)
0620 21:50:46.267631 (+ 6313us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.267653 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.267962 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.269757 (+ 1795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.269760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.283466 (+ 13706us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.283644 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.283658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.284939 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.284943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.285893 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.285908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.286693 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.286700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.286827 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.287511 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.287525 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.288004 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:46.290176 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.290179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.290182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.290436 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.291093 (+ 657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.291096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.291098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.291225 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.291685 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:46.291688 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:46.291802 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.291905 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.292069 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6182,"thread_start_us":98,"threads_started":1}
I20260620 21:50:46.292461 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.261058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.0.193.254:44219)
0620 21:50:46.261215 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.261219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.267753 (+ 6534us) server_negotiation.cc:407] Connection header received
0620 21:50:46.268146 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.268152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.268222 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.268668 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.283879 (+ 15211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.284747 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.286068 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.286532 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.286671 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.288172 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.288177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.288180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.288211 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.290040 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.290573 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.290576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.290579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.290675 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.291329 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.291333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.291335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.291579 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.291682 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.292057 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.292300 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992244, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.254@KRBTEST.COM for oryx/127.0.193.252@KRBTEST.COM
I20260620 21:50:46.308151 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.297593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:42751 (local address 127.0.0.1:34126)
0620 21:50:46.297772 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.297786 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.297868 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.298188 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.298191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.298412 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.298524 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.298531 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.299599 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.299603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.300428 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.300439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.301054 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.301058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.301167 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.301758 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.301777 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.303470 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:46.306270 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.306274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.306277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.306606 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.306962 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.306965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.306967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.307075 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.307561 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:46.307564 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:46.307674 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.307796 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.307967 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 21:50:46.308151 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.297676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.0.193.252:42751)
0620 21:50:46.297798 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.297803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.297820 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:50:46.298000 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.298003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.298058 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.298149 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.298651 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.299467 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.300559 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.300926 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.301031 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.303671 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.303675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.303678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.303711 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.306018 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.306734 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.306740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.306743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.306836 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.307181 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.307208 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.307212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.307454 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.307562 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.307868 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.307993 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:50:46.309926 6894 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.193.252:42751: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ab0459286b274ed183b16dbb8b46235a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.193.252"
port: 42751
}
attrs {
promote: true
}
}
dest_uuid: "c1af966f8a6c4dfb80f698a80ca810ef"
cas_config_opid_index: 5
I20260620 21:50:46.310855 6894 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: NON_VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: true } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } }
I20260620 21:50:46.312376 7455 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ab0459286b274ed183b16dbb8b46235a (127.0.193.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: NON_VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: true } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } } }
I20260620 21:50:46.314724 7455 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.193.252:42751
I20260620 21:50:46.315670 7461 master.cc:537] Master@127.0.193.252:42751 shutting down...
I20260620 21:50:46.315809 7588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:46.317423 7588 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 1 FOLLOWER]: Refusing update from remote peer c1af966f8a6c4dfb80f698a80ca810ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 21:50:46.318828 7641 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:46.319723 7456 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: NON_VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: true } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } } }
I20260620 21:50:46.319875 7456 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: This master's current role is: LEADER
W20260620 21:50:46.321614 6879 consensus_peers.cc:487] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef -> Peer ab0459286b274ed183b16dbb8b46235a (127.0.193.252:42751): Couldn't send request to peer ab0459286b274ed183b16dbb8b46235a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 21:50:46.322455 7461 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:46.322896 7461 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:46.323107 7461 tablet_replica.cc:330] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: stopping tablet replica
I20260620 21:50:46.327550 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.320444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:38733 (local address 127.0.0.1:42732)
0620 21:50:46.321772 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.321789 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.321894 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.322433 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.322436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.322455 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:46.322551 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.322557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.323766 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.323772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.325247 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.325262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.326948 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.326951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.327064 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.327122 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.327282 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.327378 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1244}
I20260620 21:50:46.327991 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.320677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.0.193.253:38733)
0620 21:50:46.320796 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.320801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.321853 (+ 1052us) server_negotiation.cc:407] Connection header received
0620 21:50:46.322098 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.322102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.322166 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.322264 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:46.322673 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.323562 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.325431 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.326551 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.326649 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.326822 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.327786 (+ 964us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.327876 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:50:46.328688 7506 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 LEARNER]: Refusing update from remote peer c1af966f8a6c4dfb80f698a80ca810ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 21:50:46.329617 7642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: NON_VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 21:50:46.331092 7461 master.cc:552] Master@127.0.193.252:42751 shutdown complete.
I20260620 21:50:46.331275 7461 master_runner.cc:305] Clearing existing system tablet
I20260620 21:50:46.331583 7461 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:46.331826 7461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:46.333424 7506 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ab0459286b274ed183b16dbb8b46235a (127.0.193.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: NON_VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: true } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } } }
I20260620 21:50:46.335601 7461 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:50:46.336633 7647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:46.336921 7461 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:46.337054 7461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 21:50:46.337090 7461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/wal
uuid: "ab0459286b274ed183b16dbb8b46235a"
format_stamp: "Formatted at 2026-06-20 21:50:46 on dist-test-slave-kk5c"
server_key: "c2b30ab230671c2c2b9dd6c63f7a695d"
server_key_iv: "dc1b1a848e8c0772c3f9723f23de4231"
server_key_version: "encryptionkey@0"
I20260620 21:50:46.337126 7535 sys_catalog.cc:455] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1af966f8a6c4dfb80f698a80ca810ef. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: NON_VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: true } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } } }
I20260620 21:50:46.337244 7535 sys_catalog.cc:458] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: This master's current role is: LEARNER
I20260620 21:50:46.337327 7535 sys_catalog.cc:455] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: NON_VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: true } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } } }
I20260620 21:50:46.337368 7535 sys_catalog.cc:458] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: This master's current role is: LEARNER
I20260620 21:50:46.339433 7461 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:46.342470 7461 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 21:50:46.342608 7461 log.cc:1196] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 21:50:46.343003 7461 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: Deleting consensus metadata
I20260620 21:50:46.343293 7461 master_runner.cc:315] Copying system tablet from 127.0.193.254:44219
I20260620 21:50:46.344837 7461 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.193.254:44219
I20260620 21:50:46.346540 7641 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef: attempting to promote NON_VOTER a225ae546cb44aa8a305b263966c267f to VOTER
I20260620 21:50:46.347846 7641 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } }
I20260620 21:50:46.349745 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.349393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:42751 (local address 127.0.0.1:34136)
0620 21:50:46.349550 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.349645 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 21:50:46.350361 7506 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 LEARNER]: Refusing update from remote peer c1af966f8a6c4dfb80f698a80ca810ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 21:50:46.350934 7643 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 21:50:46.351567 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.351226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:42751 (local address 127.0.0.1:34150)
0620 21:50:46.351399 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.351465 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260620 21:50:46.351720 6879 consensus_peers.cc:487] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef -> Peer ab0459286b274ed183b16dbb8b46235a (127.0.193.252:42751): Couldn't send request to peer ab0459286b274ed183b16dbb8b46235a. Status: Network error: Client connection negotiation failed: client connection to 127.0.193.252:42751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 21:50:46.352749 7643 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a225ae546cb44aa8a305b263966c267f (127.0.193.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } } }
I20260620 21:50:46.354532 7643 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } } }
I20260620 21:50:46.354664 7643 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:46.355506 7455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } } }
I20260620 21:50:46.355659 7455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:46.356688 7506 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a225ae546cb44aa8a305b263966c267f (127.0.193.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } } }
I20260620 21:50:46.358708 7646 sys_catalog.cc:455] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } } }
I20260620 21:50:46.358878 7646 sys_catalog.cc:458] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:50:46.359778 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.347796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.0.193.254:44219)
0620 21:50:46.347929 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.347934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.348657 (+ 723us) server_negotiation.cc:407] Connection header received
0620 21:50:46.348897 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.348910 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.348967 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.349022 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.349703 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.350472 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.352608 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.352928 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.353014 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.354652 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.354657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.354660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.354702 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.357109 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.357793 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.357797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.357799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.357890 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.358220 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.358227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.358231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.358478 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.358556 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.359470 (+ 914us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.359584 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:50:46.359790 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.348146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.0.1:52378)
0620 21:50:46.348558 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.348574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.348770 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.349197 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.349203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.349455 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.349573 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.349581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.350776 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.350780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.351958 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.351969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.353278 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.353281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.353366 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.353970 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.353985 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.354486 (+ 501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:46.357406 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.357409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.357411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.357671 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.358008 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.358011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.358013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.358106 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.359129 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:46.359135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:46.359288 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.359402 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.359582 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":111,"threads_started":1}
I20260620 21:50:46.360668 6915 tablet_copy_service.cc:140] P c1af966f8a6c4dfb80f698a80ca810ef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ab0459286b274ed183b16dbb8b46235a ({username='oryx', principal='oryx/127.0.193.252@KRBTEST.COM'} at 127.0.0.1:52378)
I20260620 21:50:46.360803 6915 tablet_copy_service.cc:161] P c1af966f8a6c4dfb80f698a80ca810ef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ab0459286b274ed183b16dbb8b46235a at {username='oryx', principal='oryx/127.0.193.252@KRBTEST.COM'} at 127.0.0.1:52378: session id = ab0459286b274ed183b16dbb8b46235a-00000000000000000000000000000000
I20260620 21:50:46.361649 6915 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 21:50:46.362784 7220 heartbeater.cc:503] Master 127.0.193.254:44219 was elected leader, sending a full tablet report...
I20260620 21:50:46.363073 7461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 21:50:46.365724 7461 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: tablet copy: Starting download of 0 data blocks...
I20260620 21:50:46.365978 7461 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: tablet copy: Starting download of 1 WAL segments...
I20260620 21:50:46.367600 7461 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 21:50:46.371913 6915 tablet_copy_service.cc:342] P c1af966f8a6c4dfb80f698a80ca810ef: Request end of tablet copy session ab0459286b274ed183b16dbb8b46235a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.193.252@KRBTEST.COM'} at 127.0.0.1:52378
I20260620 21:50:46.372069 6915 tablet_copy_service.cc:433] P c1af966f8a6c4dfb80f698a80ca810ef: ending tablet copy session ab0459286b274ed183b16dbb8b46235a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ab0459286b274ed183b16dbb8b46235a
I20260620 21:50:46.380069 7461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:46.382057 7654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:46.384987 7657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:46.385002 7461 server_base.cc:1034] running on GCE node
W20260620 21:50:46.387979 7655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:46.388358 7461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:46.388758 7461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:46.390100 7461 hybrid_clock.cc:648] HybridClock initialized: now 1781992246390081 us; error 48 us; skew 500 ppm
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:46.391794 7461 init.cc:377] Logged in from keytab as oryx/127.0.193.252@KRBTEST.COM (short username oryx)
I20260620 21:50:46.392354 7461 webserver.cc:460] Webserver started at http://127.0.193.252:34027/ using document root <none> and password file <none>
I20260620 21:50:46.392570 7461 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:46.392632 7461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:46.393831 7461 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:46.394814 7664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:46.395062 7461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:46.395213 7461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/wal
uuid: "ab0459286b274ed183b16dbb8b46235a"
format_stamp: "Formatted at 2026-06-20 21:50:46 on dist-test-slave-kk5c"
server_key: "c2b30ab230671c2c2b9dd6c63f7a695d"
server_key_iv: "dc1b1a848e8c0772c3f9723f23de4231"
server_key_version: "encryptionkey@0"
I20260620 21:50:46.395325 7461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:46.422171 7461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:46.424520 7461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:46.430128 7461 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.252:42751
I20260620 21:50:46.430172 7716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.252:42751 every 8 connection(s)
I20260620 21:50:46.430784 7461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/master-2/data/info.pb
I20260620 21:50:46.432811 7717 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:50:46.433925 7717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: Bootstrap starting.
I20260620 21:50:46.447397 7717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:50:46.447876 7717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: Bootstrap complete.
I20260620 21:50:46.448518 7717 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } }
I20260620 21:50:46.448676 7717 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab0459286b274ed183b16dbb8b46235a, State: Initialized, Role: LEARNER
I20260620 21:50:46.448807 7717 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } }
I20260620 21:50:46.449226 7723 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } } }
I20260620 21:50:46.449347 7717 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:46.449472 7723 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [sys.catalog]: This master's current role is: LEARNER
I20260620 21:50:46.452894 7734 catalog_manager.cc:1259] Loaded cluster ID: 30b65a13dabc4460a8959069c8331c8c
I20260620 21:50:46.453037 7734 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: loading cluster ID for follower catalog manager: success
W20260620 21:50:46.453629 7639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:46.453998 7639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:46.454066 7639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 21:50:46.455240 7734 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: acquiring CA information for follower catalog manager: success
I20260620 21:50:46.456684 7734 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 21:50:46.459301 7639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:46.459376 7639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:46.459479 7639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:46.465262 7639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.193.keytab
--principal=oryx/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:41503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=43735
--tserver_master_addrs=127.0.193.254:44219,127.0.193.253:38733,127.0.193.252:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:46.466475 7639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:46.467892 7639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:46.479840 7737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:46.479933 7736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:46.479972 7739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:46.480641 7639 server_base.cc:1034] running on GCE node
I20260620 21:50:46.481117 7639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:46.481817 7639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:46.483037 7639 hybrid_clock.cc:648] HybridClock initialized: now 1781992246483006 us; error 44 us; skew 500 ppm
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:46.486461 7639 init.cc:377] Logged in from keytab as oryx/127.0.193.193@KRBTEST.COM (short username oryx)
I20260620 21:50:46.487815 7639 webserver.cc:460] Webserver started at http://127.0.193.193:43735/ using document root <none> and password file <none>
I20260620 21:50:46.488581 7639 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:46.488716 7639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:46.493445 7639 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:50:46.496297 7746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:46.498018 7639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:46.498214 7639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "0df4a56890b34a72be8f690aa2009dcf"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "950ebce2e008181e58b522ef144bd0ba"
server_key_iv: "7cf2c72e19a4b648b3c14ee2f3d3ecd0"
server_key_version: "encryptionkey@0"
I20260620 21:50:46.499035 7639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:46.514448 7639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:46.518261 7639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:46.518435 7639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:46.518994 7639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:46.519946 7639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:46.519995 7639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:46.520078 7639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:46.520119 7639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:46.530857 7639 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:41503
I20260620 21:50:46.530877 7859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:41503 every 8 connection(s)
I20260620 21:50:46.532330 7639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:46.539389 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 7639
I20260620 21:50:46.539583 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 7091
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.253@KRBTEST.COM
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.252@KRBTEST.COM
I20260620 21:50:46.548362 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:39791
--local_ip_for_outbound_sockets=127.0.193.194
--tserver_master_addrs=127.0.193.254:44219
--webserver_port=32825
--webserver_interface=127.0.193.194
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.194.keytab
--principal=oryx/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.193.254:44219,127.0.193.253:38733
--tserver_master_addrs=127.0.193.254:44219,127.0.193.253:38733,127.0.193.252:42751 with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:50:46.552807 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.536418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:34057 (local address 127.0.193.254:44219)
0620 21:50:46.536568 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.536573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.536590 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:50:46.537336 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.537339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.537396 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.537447 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.538825 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.539740 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.542605 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.542932 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.543009 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.548104 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.548108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.548110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.548136 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.550805 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.551468 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.551471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.551473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.551555 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.551872 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.551875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.551877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.552077 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.552170 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.552511 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.552633 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 21:50:46.553324 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.534590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:50375 (local address 127.0.193.252:42751)
0620 21:50:46.535079 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.535083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.536069 (+ 986us) server_negotiation.cc:407] Connection header received
0620 21:50:46.537280 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.537285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.537347 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.537409 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.538825 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.539936 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.542189 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.542653 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.542795 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.549363 (+ 6568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.549370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.549375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.549412 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.551251 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.551879 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.551885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.551890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.551979 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.552297 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.552303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.552307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.552551 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.552628 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.553055 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.553181 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":83,"threads_started":1}
I20260620 21:50:46.553718 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.535283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:42751 (local address 127.0.193.193:50375)
0620 21:50:46.535904 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.535966 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.537040 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.537660 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.537668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.538130 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.538668 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.538675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.540115 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.540121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.542039 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.542052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.543011 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.543018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.543403 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.544653 (+ 1250us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.544671 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.549140 (+ 4469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:46.551499 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.551501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.551503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.551762 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.552092 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.552098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.552103 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.552191 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.552703 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:46.552709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:46.552834 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.552993 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.553133 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":376,"spinlock_wait_cycles":13056,"thread_start_us":144,"threads_started":1}
I20260620 21:50:46.553717 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.536353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:38733 (local address 127.0.193.193:42709)
0620 21:50:46.536603 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.536616 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.537053 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.537660 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.537667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.538077 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.538656 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.538672 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.540392 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.540399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.542284 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.542291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.543011 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.543014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.543394 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.544525 (+ 1131us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.544556 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.547691 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:46.550096 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.550103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.550125 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.550563 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.550899 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.550903 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.550908 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.551106 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.551694 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:46.551701 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:46.552053 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.552277 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.552686 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":59,"spinlock_wait_cycles":14848,"thread_start_us":99,"threads_started":1}
I20260620 21:50:46.554395 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.535396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.193:34057)
0620 21:50:46.535904 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.535966 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.537040 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.537660 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.537668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.538130 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.538665 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.538675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.540161 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.540166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.542495 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.542500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.543172 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.543176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.543405 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.544525 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.544554 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.547124 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:46.551035 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.551039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.551041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.551353 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.551676 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.551679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.551681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.551766 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.552211 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:46.552215 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:46.552309 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.552385 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.554241 (+ 1856us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":2633,"spinlock_wait_cycles":96000,"thread_start_us":156,"threads_started":1}
I20260620 21:50:46.555086 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.534590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:42709 (local address 127.0.193.253:38733)
0620 21:50:46.534734 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.534740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.536708 (+ 1968us) server_negotiation.cc:407] Connection header received
0620 21:50:46.537311 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.537315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.537378 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.537474 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.538844 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.539654 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.542399 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.542746 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.542843 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.547917 (+ 5074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.547922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.547925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.547961 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.549936 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.550691 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.550697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.550701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.550790 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.551271 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.551276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.551279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.551551 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.551647 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.554689 (+ 3042us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.554917 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:50:46.555562 7860 heartbeater.cc:347] Connected to a master server at 127.0.193.252:42751
I20260620 21:50:46.555593 7861 heartbeater.cc:347] Connected to a master server at 127.0.193.254:44219
I20260620 21:50:46.555786 7860 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:46.555788 7861 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:46.555943 7862 heartbeater.cc:347] Connected to a master server at 127.0.193.253:38733
I20260620 21:50:46.556017 7862 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:46.556468 7862 heartbeater.cc:511] Master 127.0.193.253:38733 requested a full tablet report, sending...
I20260620 21:50:46.556481 7861 heartbeater.cc:511] Master 127.0.193.254:44219 requested a full tablet report, sending...
I20260620 21:50:46.556468 7860 heartbeater.cc:511] Master 127.0.193.252:42751 requested a full tablet report, sending...
I20260620 21:50:46.557803 6894 ts_manager.cc:194] Re-registered known tserver with Master: 0df4a56890b34a72be8f690aa2009dcf (127.0.193.193:41503)
I20260620 21:50:46.558089 7496 ts_manager.cc:194] Registered new tserver with Master: 0df4a56890b34a72be8f690aa2009dcf (127.0.193.193:41503)
I20260620 21:50:46.558089 7675 ts_manager.cc:194] Registered new tserver with Master: 0df4a56890b34a72be8f690aa2009dcf (127.0.193.193:41503)
I20260620 21:50:46.558524 6894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:34057
I20260620 21:50:46.586048 7354 heartbeater.cc:503] Master 127.0.193.254:44219 was elected leader, sending a full tablet report...
W20260620 21:50:46.681926 7868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:46.682247 7868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:46.682294 7868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:46.686596 7868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:46.686678 7868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:46.686743 7868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:46.692507 7868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.194.keytab
--principal=oryx/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:39791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=32825
--tserver_master_addrs=127.0.193.254:44219,127.0.193.253:38733,127.0.193.252:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:46.693646 7868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:46.694648 7868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:46.706143 7876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:46.706158 7874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:46.706152 7873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:46.706532 7868 server_base.cc:1034] running on GCE node
I20260620 21:50:46.706974 7868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:46.707763 7868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:46.708964 7868 hybrid_clock.cc:648] HybridClock initialized: now 1781992246708937 us; error 40 us; skew 500 ppm
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:46.712636 7868 init.cc:377] Logged in from keytab as oryx/127.0.193.194@KRBTEST.COM (short username oryx)
I20260620 21:50:46.714416 7868 webserver.cc:460] Webserver started at http://127.0.193.194:32825/ using document root <none> and password file <none>
I20260620 21:50:46.715447 7868 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:46.715572 7868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:46.722523 7868 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 21:50:46.726441 7883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:46.728334 7868 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:50:46.728523 7868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "3738bd4c059c46f1985c0584596a312f"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "ffb78a1468948f442dcef564175c5486"
server_key_iv: "10a39007261ec22cde8a992d8ded420b"
server_key_version: "encryptionkey@0"
I20260620 21:50:46.729584 7868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:46.753291 7868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:46.758062 7868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:46.758329 7868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:46.759169 7868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:46.760577 7868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:46.760649 7868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:46.760720 7868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:46.760753 7868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:46.773666 7868 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:39791
I20260620 21:50:46.773715 7996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:39791 every 8 connection(s)
I20260620 21:50:46.775013 7868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:46.777779 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 7868
I20260620 21:50:46.777942 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 7225
I20260620 21:50:46.788101 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:41471
--local_ip_for_outbound_sockets=127.0.193.195
--tserver_master_addrs=127.0.193.254:44219
--webserver_port=45271
--webserver_interface=127.0.193.195
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.195.keytab
--principal=oryx/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.193.254:44219,127.0.193.253:38733
--tserver_master_addrs=127.0.193.254:44219,127.0.193.253:38733,127.0.193.252:42751 with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.253@KRBTEST.COM
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.252@KRBTEST.COM
I20260620 21:50:46.796053 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.783406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:49515 (local address 127.0.193.252:42751)
0620 21:50:46.784288 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.784292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.784303 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:50:46.784345 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.784347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.784386 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.784481 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.785272 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.786085 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.787308 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.787672 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.787771 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.791502 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.791507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.791511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.791542 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.793399 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.794139 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.794143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.794145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.794232 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.794641 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.794645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.794647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.794885 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.794951 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.795757 (+ 806us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.795895 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802}
I20260620 21:50:46.796165 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.777454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:57945 (local address 127.0.193.253:38733)
0620 21:50:46.777598 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.777602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.779624 (+ 2022us) server_negotiation.cc:407] Connection header received
0620 21:50:46.781530 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.781537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.781607 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.781668 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.783413 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.784276 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.786296 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.786685 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.786782 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.790940 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.790946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.790952 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.790986 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.793308 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.794165 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.794171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.794176 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.794284 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.794666 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.794672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.794676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.794957 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.795053 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.795784 (+ 731us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.795925 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:50:46.797894 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.778202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:38733 (local address 127.0.193.194:57945)
0620 21:50:46.779408 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.779478 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.781240 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.781968 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.781978 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.782557 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.783221 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.783246 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.784468 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.784474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.786141 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.786155 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.786962 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.786968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.787445 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.788851 (+ 1406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.788873 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.790718 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:46.793454 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.793460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.793463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.794031 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.794410 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.794415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.794419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.794545 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.795120 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:46.795125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:46.795471 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.796384 (+ 913us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.796862 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":222,"spinlock_wait_cycles":896,"thread_start_us":630,"threads_started":1}
I20260620 21:50:46.797894 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.779681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:42751 (local address 127.0.193.194:49515)
0620 21:50:46.779944 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.779960 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.781274 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.784497 (+ 3223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.784500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.784738 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.785106 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.785115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.786235 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.786239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.787109 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.787117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.787987 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.787993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.788104 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.789893 (+ 1789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.789908 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.791337 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:46.793505 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.793511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.793513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.794016 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.794353 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.794355 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.794357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.794534 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.795028 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:46.795035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:46.795466 (+ 431us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.796320 (+ 854us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.796757 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":157,"thread_start_us":125,"threads_started":1}
I20260620 21:50:46.799327 7997 heartbeater.cc:347] Connected to a master server at 127.0.193.252:42751
I20260620 21:50:46.799568 7997 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:46.800082 7997 heartbeater.cc:511] Master 127.0.193.252:42751 requested a full tablet report, sending...
I20260620 21:50:46.801250 7675 ts_manager.cc:194] Registered new tserver with Master: 3738bd4c059c46f1985c0584596a312f (127.0.193.194:39791)
I20260620 21:50:46.803269 7999 heartbeater.cc:347] Connected to a master server at 127.0.193.253:38733
I20260620 21:50:46.803365 7999 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:46.803589 7999 heartbeater.cc:511] Master 127.0.193.253:38733 requested a full tablet report, sending...
I20260620 21:50:46.804432 7496 ts_manager.cc:194] Registered new tserver with Master: 3738bd4c059c46f1985c0584596a312f (127.0.193.194:39791)
I20260620 21:50:46.805126 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.780541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:52173 (local address 127.0.193.254:44219)
0620 21:50:46.780676 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.780680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.780693 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:50:46.781532 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.781535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.781604 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.781659 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.783412 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.784099 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.786281 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.786618 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.786696 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.790926 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.790930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.790935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.790969 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.793245 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.794141 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.794144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.794146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.794259 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:46.804161 (+ 9902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:46.804166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:46.804167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:46.804405 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:46.804485 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.804833 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.804973 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:50:46.805891 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.780097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.194:52173)
0620 21:50:46.780358 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.780376 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.781274 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.781965 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.781973 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.782549 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.783179 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.783248 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.784634 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.784637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.786137 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.786149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.786903 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.786907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.787232 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.787936 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.787958 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.790719 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:46.793410 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.793417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.793439 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.794025 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.803899 (+ 9874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:46.803902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:46.803905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:46.804031 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:46.804560 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:46.804563 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:46.804654 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.805257 (+ 603us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.805418 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":346,"spinlock_wait_cycles":22784,"thread_start_us":105,"threads_started":1}
I20260620 21:50:46.806569 7998 heartbeater.cc:347] Connected to a master server at 127.0.193.254:44219
I20260620 21:50:46.806665 7998 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:46.806854 7998 heartbeater.cc:511] Master 127.0.193.254:44219 requested a full tablet report, sending...
I20260620 21:50:46.807520 6894 ts_manager.cc:194] Re-registered known tserver with Master: 3738bd4c059c46f1985c0584596a312f (127.0.193.194:39791)
I20260620 21:50:46.808373 6894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:52173
I20260620 21:50:46.858680 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.854309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:42751 (local address 127.0.0.1:34156)
0620 21:50:46.854640 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.854654 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.854760 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.855336 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.855339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.855363 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:46.855469 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.855475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.856525 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.856528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.857348 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.857359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.858313 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.858316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.858391 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.858414 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:50:46.858474 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 21:50:46.858546 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":75,"threads_started":1}
I20260620 21:50:46.858702 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.854443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34156 (local address 127.0.193.252:42751)
0620 21:50:46.854604 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.854609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.854732 (+ 123us) server_negotiation.cc:407] Connection header received
0620 21:50:46.854977 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.854981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.855049 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.855325 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:46.855605 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.856396 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.857489 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.858152 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.858275 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.858422 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:46.858492 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 21:50:46.858566 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 21:50:46.859381 7688 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 21:50:46.859787 7723 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1af966f8a6c4dfb80f698a80ca810ef. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } } }
I20260620 21:50:46.859962 7723 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [sys.catalog]: This master's current role is: LEARNER
W20260620 21:50:46.913291 8004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:46.913589 8004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:46.913633 8004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:46.918296 8004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:46.918377 8004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:46.918442 8004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:46.924532 8004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/krb5kdc/oryx_127.0.193.195.keytab
--principal=oryx/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:41471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=45271
--tserver_master_addrs=127.0.193.254:44219,127.0.193.253:38733,127.0.193.252:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:46.925671 8004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:46.926651 8004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:46.937312 8010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:46.937342 8013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:46.937400 8004 server_base.cc:1034] running on GCE node
W20260620 21:50:46.937288 8011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:46.937981 8004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:46.938750 8004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:46.939924 8004 hybrid_clock.cc:648] HybridClock initialized: now 1781992246939877 us; error 62 us; skew 500 ppm
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:46.942900 8004 init.cc:377] Logged in from keytab as oryx/127.0.193.195@KRBTEST.COM (short username oryx)
I20260620 21:50:46.943900 8004 webserver.cc:460] Webserver started at http://127.0.193.195:45271/ using document root <none> and password file <none>
I20260620 21:50:46.944489 8004 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:46.944605 8004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:46.948820 8004 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:50:46.951246 8020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:46.952600 8004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:46.952771 8004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "9fb7ce4170754c97b58212aa70323c9f"
format_stamp: "Formatted at 2026-06-20 21:50:45 on dist-test-slave-kk5c"
server_key: "03e5bb1d045e95eaed4ea47ca6f8154f"
server_key_iv: "ef66eaeb749a9d20f9be3e0c31893bad"
server_key_version: "encryptionkey@0"
I20260620 21:50:46.953362 8004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:46.968924 8004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:46.972362 8004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:46.972623 8004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:46.973240 8004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:46.974135 8004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:46.974206 8004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:46.974282 8004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:46.974331 8004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:46.983776 8004 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:41471
I20260620 21:50:46.983798 8133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:41471 every 8 connection(s)
I20260620 21:50:46.984843 8004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:50:46.986071 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 8004
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.253@KRBTEST.COM
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.252@KRBTEST.COM
Jun 20 21:50:46 dist-test-slave-kk5c krb5kdc[6850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992246, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
I20260620 21:50:47.003445 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.988652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.0.193.252:42751)
0620 21:50:46.988910 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.988914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.989937 (+ 1023us) server_negotiation.cc:407] Connection header received
0620 21:50:46.990233 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.990237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.990295 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.990365 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.991467 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.992329 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.993522 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.993885 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.993965 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.996419 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:46.996423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:46.996427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:46.996459 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:46.999122 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.000317 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.000320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.000322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.000419 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.001770 (+ 1351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.001775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.001777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.002034 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:47.002170 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.003011 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.003215 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"spinlock_wait_cycles":350848,"thread_start_us":90,"threads_started":1}
I20260620 21:50:47.006385 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.990089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.0.193.253:38733)
0620 21:50:46.990896 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.990903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.991236 (+ 333us) server_negotiation.cc:407] Connection header received
0620 21:50:46.991409 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.991415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.991467 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.991538 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.992453 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.993274 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.995326 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.995866 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.995975 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.000545 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:47.000549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:47.000551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:47.000578 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:47.002928 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.004816 (+ 1888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.004822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.004825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.004908 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.005263 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.005266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.005268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.005486 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:47.005606 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.005989 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.006139 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":71,"threads_started":1}
I20260620 21:50:47.006532 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.988881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:49173 (local address 127.0.193.253:38733)
0620 21:50:46.989021 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.989028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.989650 (+ 622us) server_negotiation.cc:407] Connection header received
0620 21:50:46.990771 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.990774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.990820 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.990892 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.993315 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.994056 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.996222 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.996550 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.996611 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.000226 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:47.000233 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:47.000239 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:47.000277 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:47.004199 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.004707 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.004714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.004721 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.004814 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.005166 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.005171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.005176 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.005427 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:47.005519 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.006291 (+ 772us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.006400 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 21:50:47.006616 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.986955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:43225 (local address 127.0.193.252:42751)
0620 21:50:46.988736 (+ 1781us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.988740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.989161 (+ 421us) server_negotiation.cc:407] Connection header received
0620 21:50:46.990349 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.990353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.990430 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.990480 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.992443 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.993179 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.997763 (+ 4584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.998107 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.998220 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.001459 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:47.001465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:47.001470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:47.001503 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:47.003684 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.004528 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.004533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.004538 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.004632 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.005110 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.005114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.005116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.005391 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:47.005482 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.006315 (+ 833us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.006447 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1679,"spinlock_wait_cycles":1408}
I20260620 21:50:47.008646 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.988249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:38733 (local address 127.0.193.195:49173)
0620 21:50:46.989564 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.989577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.990639 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.991006 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.991008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.991520 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.992282 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.992303 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.994382 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.994386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.995954 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.996029 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.996784 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.996787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.997023 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.997673 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.997700 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:46.999994 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:47.004320 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:47.004323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:47.004325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:47.004575 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:47.004941 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:47.004946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:47.004950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:47.005044 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:47.005583 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:47.005587 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:47.005990 (+ 403us) client_negotiation.cc:769] Sending connection context
0620 21:50:47.006741 (+ 751us) client_negotiation.cc:240] Negotiation successful
0620 21:50:47.007280 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1256,"mutex_wait_us":92,"spinlock_wait_cycles":2688,"thread_start_us":101,"threads_started":1}
I20260620 21:50:47.009617 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.989347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.0.193.254:44219)
0620 21:50:46.989559 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.989573 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.989950 (+ 377us) server_negotiation.cc:407] Connection header received
0620 21:50:46.991237 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.991240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.991294 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.991359 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.993102 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.993930 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.999516 (+ 5586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.999909 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.999993 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.001884 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:47.001891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:47.001894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:47.001923 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:47.007294 (+ 5371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.007991 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.007995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.007997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.008105 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.008460 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.008464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.008466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.008712 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:47.008871 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.009286 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.009433 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"spinlock_wait_cycles":1280,"thread_start_us":77,"threads_started":1}
I20260620 21:50:47.010473 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.987467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:42751 (local address 127.0.193.195:43225)
0620 21:50:46.989083 (+ 1616us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.989096 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.990115 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.991069 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.991099 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.992160 (+ 1061us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.992289 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.992303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.996836 (+ 4533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.996838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.997652 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.997664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.998261 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.998265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.998392 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.998935 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.998954 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:47.001268 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:47.003861 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:47.003868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:47.003891 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:47.004332 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:47.004760 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:47.004766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:47.004772 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:47.004991 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:47.005562 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:47.005570 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:47.005990 (+ 420us) client_negotiation.cc:769] Sending connection context
0620 21:50:47.006780 (+ 790us) client_negotiation.cc:240] Negotiation successful
0620 21:50:47.010262 (+ 3482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":3113,"spinlock_wait_cycles":13952,"thread_start_us":740,"threads_started":1}
I20260620 21:50:47.010529 8136 heartbeater.cc:347] Connected to a master server at 127.0.193.253:38733
I20260620 21:50:47.010787 8136 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:47.011096 8134 heartbeater.cc:347] Connected to a master server at 127.0.193.252:42751
I20260620 21:50:47.011179 8134 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:47.011401 8136 heartbeater.cc:511] Master 127.0.193.253:38733 requested a full tablet report, sending...
I20260620 21:50:47.012079 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.987467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.195:35145)
0620 21:50:46.988896 (+ 1429us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:46.988962 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:46.990115 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:46.990898 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:46.990909 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:46.992548 (+ 1639us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:46.992658 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.992664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.995136 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.995139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.996014 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:46.996023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.996784 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.996787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:46.997430 (+ 643us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:46.999359 (+ 1929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:46.999877 (+ 518us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:47.002789 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:47.007583 (+ 4794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:47.007586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:47.007588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:47.007843 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:47.008864 (+ 1021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:47.008867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:47.008869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:47.008957 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:47.009519 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:47.009522 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:47.010777 (+ 1255us) client_negotiation.cc:769] Sending connection context
0620 21:50:47.011728 (+ 951us) client_negotiation.cc:240] Negotiation successful
0620 21:50:47.011901 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1220,"mutex_wait_us":591,"spinlock_wait_cycles":70016,"thread_start_us":187,"threads_started":1}
I20260620 21:50:47.012085 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:46.987082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:35145 (local address 127.0.193.254:44219)
0620 21:50:46.988872 (+ 1790us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:46.988877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:46.989090 (+ 213us) server_negotiation.cc:407] Connection header received
0620 21:50:46.990349 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:46.990353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:46.990405 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:46.990462 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:46.994060 (+ 3598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.994989 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.996156 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:46.996483 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:46.996556 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.004271 (+ 7715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:47.004275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:47.004278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:47.004312 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:47.007364 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.008533 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.008537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.008539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.008649 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.009059 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.009062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.009064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.009245 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:47.009315 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.011822 (+ 2507us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.011940 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1295,"server-negotiator.queue_time_us":1678,"spinlock_wait_cycles":1792}
I20260620 21:50:47.011400 8134 heartbeater.cc:511] Master 127.0.193.252:42751 requested a full tablet report, sending...
I20260620 21:50:47.013025 7496 ts_manager.cc:194] Registered new tserver with Master: 9fb7ce4170754c97b58212aa70323c9f (127.0.193.195:41471)
I20260620 21:50:47.013027 7681 ts_manager.cc:194] Registered new tserver with Master: 9fb7ce4170754c97b58212aa70323c9f (127.0.193.195:41471)
I20260620 21:50:47.013149 8135 heartbeater.cc:347] Connected to a master server at 127.0.193.254:44219
I20260620 21:50:47.013223 8135 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:47.013394 8135 heartbeater.cc:511] Master 127.0.193.254:44219 requested a full tablet report, sending...
I20260620 21:50:47.014061 6894 ts_manager.cc:194] Re-registered known tserver with Master: 9fb7ce4170754c97b58212aa70323c9f (127.0.193.195:41471)
I20260620 21:50:47.014755 6894 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:35145
I20260620 21:50:47.014551 6895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:50:47.018258 6895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:47.021039 8153 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: NON_VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: true } } }
I20260620 21:50:47.021204 8153 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:47.024855 8152 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef: attempting to promote NON_VOTER ab0459286b274ed183b16dbb8b46235a to VOTER
I20260620 21:50:47.025377 8152 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } }
I20260620 21:50:47.026666 7506 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 FOLLOWER]: Refusing update from remote peer c1af966f8a6c4dfb80f698a80ca810ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 21:50:47.027158 8152 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 21:50:47.027937 7691 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 1 LEARNER]: Refusing update from remote peer c1af966f8a6c4dfb80f698a80ca810ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 21:50:47.028746 7641 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 21:50:47.029183 8153 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ab0459286b274ed183b16dbb8b46235a (127.0.193.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } } }
I20260620 21:50:47.029645 7506 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ab0459286b274ed183b16dbb8b46235a (127.0.193.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } } }
I20260620 21:50:47.030846 7641 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } } }
I20260620 21:50:47.030963 7641 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:47.030962 8153 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } } }
I20260620 21:50:47.031090 8153 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:47.031288 7641 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } } }
I20260620 21:50:47.031386 7641 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:47.031335 7688 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ab0459286b274ed183b16dbb8b46235a (127.0.193.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } } }
I20260620 21:50:47.033300 7723 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } } }
I20260620 21:50:47.033432 7723 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:50:47.034642 8161 sys_catalog.cc:455] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } } }
I20260620 21:50:47.034787 8161 sys_catalog.cc:458] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:50:47.039863 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.033041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:41503 (local address 127.0.0.1:37862)
0620 21:50:47.033354 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:47.033371 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:47.033451 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:47.035883 (+ 2432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:47.035886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:47.035904 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:47.036001 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:47.036007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.037091 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.037094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:47.038279 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:47.038289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.039460 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.039463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:47.039551 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.039578 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:50:47.039624 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 21:50:47.039707 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":30}
I20260620 21:50:47.040091 8164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.033753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.0.193.194:39791)
0620 21:50:47.035743 (+ 1990us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:47.035756 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:47.035780 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:50:47.035873 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:47.035878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:47.036062 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:47.036214 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:47.036407 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.037530 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.038850 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.039509 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.039634 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.039793 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.039848 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.039937 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1876,"thread_start_us":142,"threads_started":1}
I20260620 21:50:47.040159 8165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.033334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:39791 (local address 127.0.0.1:50412)
0620 21:50:47.035337 (+ 2003us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:47.035354 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:47.035455 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:47.036186 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:47.036189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:47.036205 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:47.036282 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:47.036287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.037681 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.037684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:47.038723 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:47.038730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.039643 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.039646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:47.039729 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.039750 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:50:47.039815 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 21:50:47.039882 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1932,"mutex_wait_us":7,"spinlock_wait_cycles":6912,"thread_start_us":116,"threads_started":1}
I20260620 21:50:47.040545 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.033638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.0.193.193:41503)
0620 21:50:47.035390 (+ 1752us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:47.035403 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:47.035425 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:50:47.035494 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:47.035498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:47.035758 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:47.035888 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:47.036091 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.036997 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.038418 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.039326 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.040078 (+ 752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.040253 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.040311 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.040404 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1650,"thread_start_us":90,"threads_started":1}
I20260620 21:50:47.042682 8162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.034045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:41471 (local address 127.0.0.1:53088)
0620 21:50:47.035337 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:47.035354 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:47.035455 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:47.037195 (+ 1740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:47.037198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:47.037217 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:47.037303 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:47.037308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.038505 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.038509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:47.039659 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:47.039665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.042252 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.042256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:47.042348 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.042372 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:50:47.042456 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:50:47.042537 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1241,"mutex_wait_us":3,"spinlock_wait_cycles":2176,"thread_start_us":95,"threads_started":1}
I20260620 21:50:47.042698 8166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.034188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.0.193.195:41471)
0620 21:50:47.036509 (+ 2321us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:47.036520 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:47.036541 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:50:47.036608 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:47.036613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:47.036783 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:47.036934 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:47.037437 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.038334 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.039793 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.042107 (+ 2314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.042245 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.042382 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.042484 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.042574 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2215,"thread_start_us":112,"threads_started":1}
I20260620 21:50:47.043481 7931 tablet_service.cc:1467] Processing CreateTablet for tablet 52a9bbaf810a4f5082b65c294a2aaac4 (DEFAULT_TABLE table=test-table [id=01cd9c8e226a46a3aea29ab29f4d3839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:47.043478 7794 tablet_service.cc:1467] Processing CreateTablet for tablet 52a9bbaf810a4f5082b65c294a2aaac4 (DEFAULT_TABLE table=test-table [id=01cd9c8e226a46a3aea29ab29f4d3839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:47.044620 7794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a9bbaf810a4f5082b65c294a2aaac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:47.044626 7931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a9bbaf810a4f5082b65c294a2aaac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:47.045148 8068 tablet_service.cc:1467] Processing CreateTablet for tablet 52a9bbaf810a4f5082b65c294a2aaac4 (DEFAULT_TABLE table=test-table [id=01cd9c8e226a46a3aea29ab29f4d3839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:47.046193 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a9bbaf810a4f5082b65c294a2aaac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:47.053103 8167 tablet_bootstrap.cc:492] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f: Bootstrap starting.
I20260620 21:50:47.054425 8168 tablet_bootstrap.cc:492] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf: Bootstrap starting.
I20260620 21:50:47.054952 8169 tablet_bootstrap.cc:492] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f: Bootstrap starting.
I20260620 21:50:47.055459 8167 tablet_bootstrap.cc:654] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:47.056349 8167 log.cc:824] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:47.057073 8169 tablet_bootstrap.cc:654] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:47.057073 8168 tablet_bootstrap.cc:654] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:47.057910 8168 log.cc:824] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:47.057910 8169 log.cc:824] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:47.058781 8167 tablet_bootstrap.cc:492] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f: No bootstrap required, opened a new log
I20260620 21:50:47.058984 8167 ts_tablet_manager.cc:1397] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:50:47.059688 8169 tablet_bootstrap.cc:492] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f: No bootstrap required, opened a new log
I20260620 21:50:47.059882 8169 ts_tablet_manager.cc:1397] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:50:47.059954 8168 tablet_bootstrap.cc:492] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf: No bootstrap required, opened a new log
I20260620 21:50:47.060154 8168 ts_tablet_manager.cc:1397] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:50:47.063333 8169 raft_consensus.cc:348] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } } peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } }
I20260620 21:50:47.063654 8169 raft_consensus.cc:374] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:47.063715 8169 raft_consensus.cc:729] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3738bd4c059c46f1985c0584596a312f, State: Initialized, Role: FOLLOWER
I20260620 21:50:47.064247 8169 consensus_queue.cc:260] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } } peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } }
I20260620 21:50:47.064138 8167 raft_consensus.cc:348] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } } peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } }
I20260620 21:50:47.064388 8167 raft_consensus.cc:374] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:47.064446 8167 raft_consensus.cc:729] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb7ce4170754c97b58212aa70323c9f, State: Initialized, Role: FOLLOWER
I20260620 21:50:47.064445 8168 raft_consensus.cc:348] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } } peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } }
I20260620 21:50:47.064692 8168 raft_consensus.cc:374] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:47.064754 8168 raft_consensus.cc:729] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df4a56890b34a72be8f690aa2009dcf, State: Initialized, Role: FOLLOWER
I20260620 21:50:47.064946 8167 consensus_queue.cc:260] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } } peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } }
I20260620 21:50:47.065138 8168 consensus_queue.cc:260] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } } peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } }
I20260620 21:50:47.065326 8169 ts_tablet_manager.cc:1428] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 21:50:47.065500 7998 heartbeater.cc:503] Master 127.0.193.254:44219 was elected leader, sending a full tablet report...
I20260620 21:50:47.065908 8167 ts_tablet_manager.cc:1428] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 21:50:47.065908 8168 ts_tablet_manager.cc:1428] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:50:47.066260 7861 heartbeater.cc:503] Master 127.0.193.254:44219 was elected leader, sending a full tablet report...
I20260620 21:50:47.067785 8135 heartbeater.cc:503] Master 127.0.193.254:44219 was elected leader, sending a full tablet report...
I20260620 21:50:47.187769 8174 raft_consensus.cc:482] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:47.187952 8174 raft_consensus.cc:504] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } } peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } }
I20260620 21:50:47.189215 8174 leader_election.cc:288] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0df4a56890b34a72be8f690aa2009dcf (127.0.193.193:41503), 3738bd4c059c46f1985c0584596a312f (127.0.193.194:39791)
I20260620 21:50:47.194770 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.189594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:48469 (local address 127.0.193.193:41503)
0620 21:50:47.189764 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:47.189770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:47.189792 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:50:47.189853 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:47.189856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:47.189906 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:47.189991 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:47.190428 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.191281 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.193204 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.193941 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.194083 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.194139 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.194550 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.194633 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 21:50:47.194770 8164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.189569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:51733 (local address 127.0.193.194:39791)
0620 21:50:47.189698 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:47.189703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:47.189805 (+ 102us) server_negotiation.cc:407] Connection header received
0620 21:50:47.189869 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:47.189872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:47.189914 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:47.189990 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:47.190428 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.191248 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.193196 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.193942 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.194083 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.194139 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.194550 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.194633 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:50:47.194800 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.189494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:39791 (local address 127.0.193.195:51733)
0620 21:50:47.189715 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:47.189729 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:47.189806 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:47.190020 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:47.190023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:47.190050 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:47.190285 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:47.190295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.191411 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.191415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:47.192982 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:47.192994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.194165 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.194168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:47.194346 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.194416 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 21:50:47.194504 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 21:50:47.194610 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":245,"spinlock_wait_cycles":5120}
I20260620 21:50:47.194814 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.189405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:41503 (local address 127.0.193.195:48469)
0620 21:50:47.189537 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:47.189552 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:47.189640 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:47.190020 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:47.190023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:47.190051 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:47.190246 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:47.190253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.191411 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.191415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:47.192982 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:47.192994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.194118 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.194123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:47.194318 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.194399 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 21:50:47.194504 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 21:50:47.194626 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":294,"spinlock_wait_cycles":6656}
I20260620 21:50:47.195629 7814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a9bbaf810a4f5082b65c294a2aaac4" candidate_uuid: "9fb7ce4170754c97b58212aa70323c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df4a56890b34a72be8f690aa2009dcf" is_pre_election: true
I20260620 21:50:47.195650 7951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a9bbaf810a4f5082b65c294a2aaac4" candidate_uuid: "9fb7ce4170754c97b58212aa70323c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3738bd4c059c46f1985c0584596a312f" is_pre_election: true
I20260620 21:50:47.195974 7951 raft_consensus.cc:2454] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fb7ce4170754c97b58212aa70323c9f in term 0.
I20260620 21:50:47.195995 7814 raft_consensus.cc:2454] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fb7ce4170754c97b58212aa70323c9f in term 0.
I20260620 21:50:47.196506 8024 leader_election.cc:302] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0df4a56890b34a72be8f690aa2009dcf, 3738bd4c059c46f1985c0584596a312f, 9fb7ce4170754c97b58212aa70323c9f; no voters:
I20260620 21:50:47.196836 8174 raft_consensus.cc:2781] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:47.196923 8174 raft_consensus.cc:482] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:47.196959 8174 raft_consensus.cc:3037] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:47.198043 8174 raft_consensus.cc:504] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } } peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } }
I20260620 21:50:47.198421 8174 leader_election.cc:288] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [CANDIDATE]: Term 1 election: Requested vote from peers 0df4a56890b34a72be8f690aa2009dcf (127.0.193.193:41503), 3738bd4c059c46f1985c0584596a312f (127.0.193.194:39791)
I20260620 21:50:47.198843 7951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a9bbaf810a4f5082b65c294a2aaac4" candidate_uuid: "9fb7ce4170754c97b58212aa70323c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3738bd4c059c46f1985c0584596a312f"
I20260620 21:50:47.198957 7951 raft_consensus.cc:3037] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:47.199414 7814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a9bbaf810a4f5082b65c294a2aaac4" candidate_uuid: "9fb7ce4170754c97b58212aa70323c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df4a56890b34a72be8f690aa2009dcf"
I20260620 21:50:47.199525 7814 raft_consensus.cc:3037] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:47.200346 7951 raft_consensus.cc:2454] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fb7ce4170754c97b58212aa70323c9f in term 1.
I20260620 21:50:47.200706 8023 leader_election.cc:302] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3738bd4c059c46f1985c0584596a312f, 9fb7ce4170754c97b58212aa70323c9f; no voters:
I20260620 21:50:47.200870 7814 raft_consensus.cc:2454] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fb7ce4170754c97b58212aa70323c9f in term 1.
I20260620 21:50:47.200913 8174 raft_consensus.cc:2781] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:47.201180 8174 raft_consensus.cc:686] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [term 1 LEADER]: Becoming Leader. State: Replica: 9fb7ce4170754c97b58212aa70323c9f, State: Running, Role: LEADER
I20260620 21:50:47.201557 8174 consensus_queue.cc:237] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } } peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } }
I20260620 21:50:47.205358 6895 catalog_manager.cc:5495] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fb7ce4170754c97b58212aa70323c9f (127.0.193.195). New cstate: current_term: 1 leader_uuid: "9fb7ce4170754c97b58212aa70323c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } health_report { overall_health: HEALTHY } } }
I20260620 21:50:47.234465 8166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.230215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.0.193.195:41471)
0620 21:50:47.230417 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:47.230420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:47.230456 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:50:47.230539 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:47.230541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:47.230597 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:47.230683 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:47.230994 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.231656 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.232606 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.232834 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.232941 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.233368 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:47.233941 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:47.234051 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.234187 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.234252 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260620 21:50:47.236138 8137 tablet.cc:2369] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:50:47.241083 7951 raft_consensus.cc:1261] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f [term 1 FOLLOWER]: Refusing update from remote peer 9fb7ce4170754c97b58212aa70323c9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:47.241079 7814 raft_consensus.cc:1261] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf [term 1 FOLLOWER]: Refusing update from remote peer 9fb7ce4170754c97b58212aa70323c9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:47.241933 8176 consensus_queue.cc:1035] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:47.242133 8174 consensus_queue.cc:1035] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:47.254753 8181 mvcc.cc:205] Tried to move back new op lower bound from 7299040244690935808 to 7299040244540702720. Current Snapshot: MvccSnapshot[applied={T|T < 7299040244690935808}]
I20260620 21:50:47.263932 6894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:47.264161 6894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:47.269217 6894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1af966f8a6c4dfb80f698a80ca810ef: Sending DeleteTablet for 3 replicas of tablet 52a9bbaf810a4f5082b65c294a2aaac4
I20260620 21:50:47.270284 7931 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a9bbaf810a4f5082b65c294a2aaac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:47 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:50412
I20260620 21:50:47.270296 8068 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a9bbaf810a4f5082b65c294a2aaac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:47 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:53088
I20260620 21:50:47.270323 7794 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a9bbaf810a4f5082b65c294a2aaac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:47 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:37862
I20260620 21:50:47.271471 8192 tablet_replica.cc:330] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f: stopping tablet replica
I20260620 21:50:47.271478 8191 tablet_replica.cc:330] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f: stopping tablet replica
I20260620 21:50:47.271862 8192 raft_consensus.cc:2229] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:47.271986 8191 raft_consensus.cc:2229] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:50:47.272258 8192 raft_consensus.cc:2258] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:47.272939 8191 raft_consensus.cc:2258] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:47.274266 8192 ts_tablet_manager.cc:1905] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:47.275280 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 6863
I20260620 21:50:47.279127 8192 ts_tablet_manager.cc:1918] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:47.279305 8192 log.cc:1196] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/52a9bbaf810a4f5082b65c294a2aaac4
I20260620 21:50:47.279660 8192 ts_tablet_manager.cc:1939] T 52a9bbaf810a4f5082b65c294a2aaac4 P 3738bd4c059c46f1985c0584596a312f: Deleting consensus metadata
I20260620 21:50:47.280686 8191 ts_tablet_manager.cc:1905] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:47.281343 8190 tablet_replica.cc:330] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf: stopping tablet replica
I20260620 21:50:47.281713 8190 raft_consensus.cc:2229] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:47.282074 8190 raft_consensus.cc:2258] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:47.283649 8190 ts_tablet_manager.cc:1905] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 21:50:47.284695 7884 connection.cc:535] server connection from 127.0.0.1:50412 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 21:50:47.285557 8191 ts_tablet_manager.cc:1918] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:47.285759 8191 log.cc:1196] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/52a9bbaf810a4f5082b65c294a2aaac4
I20260620 21:50:47.286196 8191 ts_tablet_manager.cc:1939] T 52a9bbaf810a4f5082b65c294a2aaac4 P 9fb7ce4170754c97b58212aa70323c9f: Deleting consensus metadata
I20260620 21:50:47.288270 8190 ts_tablet_manager.cc:1918] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:47.288461 8190 log.cc:1196] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-0/wal/wals/52a9bbaf810a4f5082b65c294a2aaac4
I20260620 21:50:47.288815 8190 ts_tablet_manager.cc:1939] T 52a9bbaf810a4f5082b65c294a2aaac4 P 0df4a56890b34a72be8f690aa2009dcf: Deleting consensus metadata
W20260620 21:50:47.289789 8024 connection.cc:441] server connection from 127.0.0.1:53088 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53088 (request call id 1) could send its response
W20260620 21:50:47.289803 7749 connection.cc:441] server connection from 127.0.0.1:37862 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37862 (request call id 1) could send its response
I20260620 21:50:47.296607 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.273250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.0.193.253:38733)
0620 21:50:47.273379 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:47.273385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:47.284118 (+ 10733us) server_negotiation.cc:407] Connection header received
0620 21:50:47.284350 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:47.284354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:47.284406 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:47.284460 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:47.286351 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.287161 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.289867 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.290223 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.290300 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.292392 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:47.292396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:47.292400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:47.292426 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:47.294685 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.295239 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.295243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.295245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.295327 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.295628 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.295631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.295633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.295851 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:47.295956 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.296313 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.296438 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:50:47.299965 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.288618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.0.193.253:38733)
0620 21:50:47.288724 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:47.288729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:47.289277 (+ 548us) server_negotiation.cc:407] Connection header received
0620 21:50:47.289468 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:47.289473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:47.289528 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:47.289581 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:47.290982 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.291804 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.292940 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.293295 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.293362 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.295087 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:47.295091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:47.295094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:47.295119 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:47.297073 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.298463 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.298469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.298471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.298570 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.298897 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.298903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.298906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.299125 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:47.299292 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.299642 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.299789 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:50:47.300204 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.288427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34202 (local address 127.0.193.252:42751)
0620 21:50:47.288573 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:47.288577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:47.289340 (+ 763us) server_negotiation.cc:407] Connection header received
0620 21:50:47.289510 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:47.289514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:47.289560 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:47.289610 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:47.291449 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.292186 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.293267 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.293592 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.293670 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.295432 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:47.295436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:47.295440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:47.295473 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:47.297309 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.298780 (+ 1471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.298785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.298786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.298873 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.299160 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.299165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.299169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.299482 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:47.299583 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.299915 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.300044 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:50:47.306967 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:47.273743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.0.193.252:42751)
0620 21:50:47.273892 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:47.273896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:47.297506 (+ 23610us) server_negotiation.cc:407] Connection header received
0620 21:50:47.297648 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:47.297651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:47.297697 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:47.297781 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:47.298550 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.299400 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.301226 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:47.301563 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:47.301631 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:47.303378 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:47.303384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:47.303390 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:47.303428 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:47.305160 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.305640 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.305644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.305645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.305728 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:47.306017 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:47.306020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:47.306022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:47.306248 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:47.306349 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:47.306693 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 21:50:47.306814 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 21:50:48.255708 8201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.255261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.195:37217)
0620 21:50:48.255495 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.255576 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
W20260620 21:50:48.256038 8135 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:44219 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
I20260620 21:50:48.257467 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.256766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.194:48121)
0620 21:50:48.257263 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.257359 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":81,"threads_started":1}
W20260620 21:50:48.257891 7998 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:44219 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
I20260620 21:50:48.258342 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.257561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.193:36967)
0620 21:50:48.258148 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.258234 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":68,"threads_started":1}
W20260620 21:50:48.258687 7861 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:44219 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
I20260620 21:50:48.770153 8204 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1af966f8a6c4dfb80f698a80ca810ef)
I20260620 21:50:48.770460 8204 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } }
I20260620 21:50:48.770751 8205 raft_consensus.cc:482] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1af966f8a6c4dfb80f698a80ca810ef)
I20260620 21:50:48.770951 8205 raft_consensus.cc:504] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } }
I20260620 21:50:48.771883 8204 leader_election.cc:288] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1af966f8a6c4dfb80f698a80ca810ef (127.0.193.254:44219), a225ae546cb44aa8a305b263966c267f (127.0.193.253:38733)
I20260620 21:50:48.772245 8205 leader_election.cc:288] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1af966f8a6c4dfb80f698a80ca810ef (127.0.193.254:44219), ab0459286b274ed183b16dbb8b46235a (127.0.193.252:42751)
I20260620 21:50:48.772764 8206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.772003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.0.1:52850)
0620 21:50:48.772561 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.772641 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":83,"threads_started":1}
I20260620 21:50:48.772774 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.772049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.0.1:52864)
0620 21:50:48.772605 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.772681 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":70,"threads_started":1}
W20260620 21:50:48.772938 7668 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
W20260620 21:50:48.772938 7483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
I20260620 21:50:48.773999 8206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.773730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.0.1:52880)
0620 21:50:48.773873 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.773919 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260620 21:50:48.774163 7668 leader_election.cc:334] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1af966f8a6c4dfb80f698a80ca810ef (127.0.193.254:44219): Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
I20260620 21:50:48.774609 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.774149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.0.1:52894)
0620 21:50:48.774385 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.774444 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
W20260620 21:50:48.775413 7483 leader_election.cc:334] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1af966f8a6c4dfb80f698a80ca810ef (127.0.193.254:44219): Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
I20260620 21:50:48.776799 8209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.772243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.0.193.252:42751)
0620 21:50:48.772448 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:48.772454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:48.772685 (+ 231us) server_negotiation.cc:407] Connection header received
0620 21:50:48.772738 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:48.772741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:48.772792 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:48.772876 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:48.773242 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.773953 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.775303 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.776197 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.776353 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.776414 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:48.776588 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 21:50:48.776661 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":204,"server-negotiator.queue_time_us":130,"thread_start_us":53,"threads_started":1}
I20260620 21:50:48.776801 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.772071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:42751 (local address 127.0.0.1:34204)
0620 21:50:48.772441 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.772457 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:48.772589 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:48.772989 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:48.772992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:48.773010 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:48.773108 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:48.773114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.774083 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.774086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:48.775146 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:48.775155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.776354 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.776358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:48.776455 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.776481 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:50:48.776573 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 21:50:48.776661 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"spinlock_wait_cycles":11008,"thread_start_us":68,"threads_started":1}
I20260620 21:50:48.777355 7688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a225ae546cb44aa8a305b263966c267f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ab0459286b274ed183b16dbb8b46235a" is_pre_election: true
I20260620 21:50:48.777585 7688 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a225ae546cb44aa8a305b263966c267f in term 1.
I20260620 21:50:48.777879 8211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.772430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.0.193.253:38733)
0620 21:50:48.772713 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:48.772718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:48.773124 (+ 406us) server_negotiation.cc:407] Connection header received
0620 21:50:48.773347 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:48.773353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:48.773403 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:48.773463 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:48.774554 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.775395 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.776834 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.777344 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.777471 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.777524 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:48.777698 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 21:50:48.777767 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":86,"threads_started":1}
I20260620 21:50:48.777880 8210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.772374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:38733 (local address 127.0.0.1:42774)
0620 21:50:48.772972 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.772987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:48.773198 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:48.773610 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:48.773612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:48.773630 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:48.774420 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:48.774430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.775527 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.775530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:48.776705 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:48.776714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.777470 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.777474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:48.777566 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.777589 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:50:48.777682 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 21:50:48.777756 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":52,"threads_started":1}
I20260620 21:50:48.777997 7480 leader_election.cc:302] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a225ae546cb44aa8a305b263966c267f, ab0459286b274ed183b16dbb8b46235a; no voters: c1af966f8a6c4dfb80f698a80ca810ef
I20260620 21:50:48.778249 8205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 21:50:48.778316 8205 raft_consensus.cc:482] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1af966f8a6c4dfb80f698a80ca810ef)
I20260620 21:50:48.778385 8205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:50:48.778317 7506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab0459286b274ed183b16dbb8b46235a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a225ae546cb44aa8a305b263966c267f" is_pre_election: true
I20260620 21:50:48.779559 8205 raft_consensus.cc:504] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } }
I20260620 21:50:48.779809 7506 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab0459286b274ed183b16dbb8b46235a in current term 2: Already voted for candidate a225ae546cb44aa8a305b263966c267f in this term.
I20260620 21:50:48.780009 8205 leader_election.cc:288] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [CANDIDATE]: Term 2 election: Requested vote from peers c1af966f8a6c4dfb80f698a80ca810ef (127.0.193.254:44219), ab0459286b274ed183b16dbb8b46235a (127.0.193.252:42751)
I20260620 21:50:48.780433 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.780152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.0.1:52898)
0620 21:50:48.780282 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.780369 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 21:50:48.780469 7688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a225ae546cb44aa8a305b263966c267f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ab0459286b274ed183b16dbb8b46235a"
I20260620 21:50:48.780521 7666 leader_election.cc:302] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab0459286b274ed183b16dbb8b46235a; no voters: a225ae546cb44aa8a305b263966c267f, c1af966f8a6c4dfb80f698a80ca810ef
I20260620 21:50:48.780602 7688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:50:48.781635 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.781385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.0.1:52908)
0620 21:50:48.781533 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.781584 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 21:50:48.781798 7483 leader_election.cc:334] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1af966f8a6c4dfb80f698a80ca810ef (127.0.193.254:44219): Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
I20260620 21:50:48.781884 7688 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a225ae546cb44aa8a305b263966c267f in term 2.
I20260620 21:50:48.782027 8204 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:50:48.782204 7480 leader_election.cc:302] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a225ae546cb44aa8a305b263966c267f, ab0459286b274ed183b16dbb8b46235a; no voters: c1af966f8a6c4dfb80f698a80ca810ef
I20260620 21:50:48.782442 8205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:50:48.782645 8205 raft_consensus.cc:686] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [term 2 LEADER]: Becoming Leader. State: Replica: a225ae546cb44aa8a305b263966c267f, State: Running, Role: LEADER
I20260620 21:50:48.782913 8205 consensus_queue.cc:237] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } }
I20260620 21:50:48.783946 8214 sys_catalog.cc:455] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a225ae546cb44aa8a305b263966c267f. Latest consensus state: current_term: 2 leader_uuid: "a225ae546cb44aa8a305b263966c267f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } } }
I20260620 21:50:48.784067 8214 sys_catalog.cc:458] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:48.784365 8216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:48.786711 8216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:48.787150 8216 catalog_manager.cc:1259] Loaded cluster ID: 30b65a13dabc4460a8959069c8331c8c
I20260620 21:50:48.787247 8216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:48.787958 8216 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:48.788363 8216 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: Loaded TSK: 0
I20260620 21:50:48.789161 8216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:50:48.816834 7495 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:50:48.820112 7495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:48.823423 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.823084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.0.1:52922)
0620 21:50:48.823259 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.823328 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 21:50:48.823673 7688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [term 2 FOLLOWER]: Refusing update from remote peer a225ae546cb44aa8a305b263966c267f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 21:50:48.824239 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.823987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.0.1:52934)
0620 21:50:48.824107 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.824166 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 21:50:48.824349 7483 consensus_peers.cc:487] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f -> Peer c1af966f8a6c4dfb80f698a80ca810ef (127.0.193.254:44219): Couldn't send request to peer c1af966f8a6c4dfb80f698a80ca810ef. Status: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 21:50:48.824386 8214 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 21:50:48.826310 8204 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a225ae546cb44aa8a305b263966c267f. Latest consensus state: current_term: 2 leader_uuid: "a225ae546cb44aa8a305b263966c267f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } } }
I20260620 21:50:48.826457 8204 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:50:48.827675 8205 sys_catalog.cc:455] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a225ae546cb44aa8a305b263966c267f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } } }
I20260620 21:50:48.827839 8205 sys_catalog.cc:458] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:48.828143 8218 mvcc.cc:205] Tried to move back new op lower bound from 7299040251175297024 to 7299040251017191424. Current Snapshot: MvccSnapshot[applied={T|T < 7299040251175297024}]
I20260620 21:50:48.828329 8204 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a225ae546cb44aa8a305b263966c267f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1af966f8a6c4dfb80f698a80ca810ef" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 44219 } } peers { permanent_uuid: "a225ae546cb44aa8a305b263966c267f" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 38733 } attrs { promote: false } } peers { permanent_uuid: "ab0459286b274ed183b16dbb8b46235a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 42751 } attrs { promote: false } } }
I20260620 21:50:48.828457 8204 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0459286b274ed183b16dbb8b46235a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:50:48.839102 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.834355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:41471 (local address 127.0.0.1:53108)
0620 21:50:48.834484 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.834520 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:48.834606 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:48.835409 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:48.835412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:48.835428 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:48.835525 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:48.835531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.836377 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.836380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:48.837694 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:48.837705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.838699 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.838703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:48.838828 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.838851 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:50:48.838910 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 21:50:48.838967 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":18}
I20260620 21:50:48.839181 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.834741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.0.193.195:41471)
0620 21:50:48.835105 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:48.835111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:48.835125 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:50:48.835234 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:48.835239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:48.835290 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:48.835399 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:48.835712 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.836236 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.837842 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.838542 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.838813 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.838878 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:48.838924 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 21:50:48.839000 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":85,"threads_started":1}
I20260620 21:50:48.839833 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.835260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:41503 (local address 127.0.0.1:37878)
0620 21:50:48.835636 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.835652 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:48.835744 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:48.836023 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:48.836026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:48.836043 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:48.836145 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:48.836151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.837111 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.837115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:48.838084 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:48.838092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.838735 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.838738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:48.839510 (+ 772us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.839533 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:50:48.839636 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 21:50:48.839705 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260620 21:50:48.839882 8068 tablet_service.cc:1467] Processing CreateTablet for tablet da6d535d1a754c48bffdea62333a9945 (DEFAULT_TABLE table=test-table [id=110aff8fe26f4d6695b6ccb1ad529034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:48.839866 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.835525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.0.193.193:41503)
0620 21:50:48.835777 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:48.835784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:48.835801 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:50:48.835854 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:48.835857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:48.835910 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:48.836016 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:48.836342 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.836997 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.838202 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.838616 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.838734 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.838804 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:48.839660 (+ 856us) server_negotiation.cc:299] Negotiation successful
0620 21:50:48.839742 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260620 21:50:48.840201 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6d535d1a754c48bffdea62333a9945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:48.840536 7794 tablet_service.cc:1467] Processing CreateTablet for tablet da6d535d1a754c48bffdea62333a9945 (DEFAULT_TABLE table=test-table [id=110aff8fe26f4d6695b6ccb1ad529034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:48.840833 7794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6d535d1a754c48bffdea62333a9945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:48.841094 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.834983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:39791 (local address 127.0.0.1:50428)
0620 21:50:48.835142 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.835161 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:48.835273 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:48.836483 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:48.836486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:48.836506 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:48.836616 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:48.836622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.838114 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.838117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:48.839325 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:48.839337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.840706 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.840710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:48.840803 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.840827 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:50:48.840877 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 21:50:48.840944 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":22}
I20260620 21:50:48.841718 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.835071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.0.193.194:39791)
0620 21:50:48.836263 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:48.836268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:48.836282 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:50:48.836334 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:48.836337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:48.836386 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:48.836475 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:48.836764 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.837969 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.839715 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.840373 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.841362 (+ 989us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.841434 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:48.841476 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 21:50:48.841553 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1098,"thread_start_us":77,"threads_started":1}
I20260620 21:50:48.842280 7931 tablet_service.cc:1467] Processing CreateTablet for tablet da6d535d1a754c48bffdea62333a9945 (DEFAULT_TABLE table=test-table [id=110aff8fe26f4d6695b6ccb1ad529034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:48.842530 7931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6d535d1a754c48bffdea62333a9945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:48.842931 8227 tablet_bootstrap.cc:492] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf: Bootstrap starting.
I20260620 21:50:48.843817 8227 tablet_bootstrap.cc:654] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:48.844427 8228 tablet_bootstrap.cc:492] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f: Bootstrap starting.
I20260620 21:50:48.844967 8227 tablet_bootstrap.cc:492] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf: No bootstrap required, opened a new log
I20260620 21:50:48.845079 8227 ts_tablet_manager.cc:1397] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:50:48.845124 8230 tablet_bootstrap.cc:492] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f: Bootstrap starting.
I20260620 21:50:48.845554 8227 raft_consensus.cc:348] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } } peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } }
I20260620 21:50:48.845680 8227 raft_consensus.cc:374] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:48.845715 8227 raft_consensus.cc:729] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df4a56890b34a72be8f690aa2009dcf, State: Initialized, Role: FOLLOWER
I20260620 21:50:48.845839 8227 consensus_queue.cc:260] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } } peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } }
I20260620 21:50:48.846038 8230 tablet_bootstrap.cc:654] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:48.846227 8227 ts_tablet_manager.cc:1428] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:48.847141 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.846557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.193:39043)
0620 21:50:48.846967 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.847032 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":98,"threads_started":1}
I20260620 21:50:48.847919 8228 tablet_bootstrap.cc:654] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:48.848300 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.847666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.193:51863)
0620 21:50:48.848174 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.848234 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437}
I20260620 21:50:48.849114 7862 heartbeater.cc:503] Master 127.0.193.253:38733 was elected leader, sending a full tablet report...
I20260620 21:50:48.849217 8230 tablet_bootstrap.cc:492] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f: No bootstrap required, opened a new log
I20260620 21:50:48.849313 8230 ts_tablet_manager.cc:1397] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 21:50:48.849822 8230 raft_consensus.cc:348] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } } peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } }
I20260620 21:50:48.849948 8230 raft_consensus.cc:374] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:48.849992 8230 raft_consensus.cc:729] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3738bd4c059c46f1985c0584596a312f, State: Initialized, Role: FOLLOWER
I20260620 21:50:48.850131 8230 consensus_queue.cc:260] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } } peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } }
I20260620 21:50:48.850471 8230 ts_tablet_manager.cc:1428] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 21:50:48.851174 8228 tablet_bootstrap.cc:492] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f: No bootstrap required, opened a new log
I20260620 21:50:48.851312 8228 ts_tablet_manager.cc:1397] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 21:50:48.851796 8228 raft_consensus.cc:348] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } } peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } }
I20260620 21:50:48.851902 8228 raft_consensus.cc:374] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:48.851936 8228 raft_consensus.cc:729] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb7ce4170754c97b58212aa70323c9f, State: Initialized, Role: FOLLOWER
I20260620 21:50:48.852062 8228 consensus_queue.cc:260] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } } peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } }
I20260620 21:50:48.852407 8228 ts_tablet_manager.cc:1428] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:48.852546 7999 heartbeater.cc:503] Master 127.0.193.253:38733 was elected leader, sending a full tablet report...
I20260620 21:50:48.853358 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.852966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.195:36069)
0620 21:50:48.853190 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.853262 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260620 21:50:48.854176 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.852253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.194:34033)
0620 21:50:48.854033 (+ 1780us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.854092 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1680,"thread_start_us":98,"threads_started":1}
I20260620 21:50:48.870289 8231 raft_consensus.cc:482] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:48.870440 8231 raft_consensus.cc:504] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } } peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } }
I20260620 21:50:48.871625 8231 leader_election.cc:288] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fb7ce4170754c97b58212aa70323c9f (127.0.193.195:41471), 3738bd4c059c46f1985c0584596a312f (127.0.193.194:39791)
I20260620 21:50:48.877157 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.871992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:39189 (local address 127.0.193.194:39791)
0620 21:50:48.872160 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:48.872165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:48.872294 (+ 129us) server_negotiation.cc:407] Connection header received
0620 21:50:48.872443 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:48.872446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:48.872493 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:48.872560 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:48.872813 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.873550 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.875527 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.876171 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.876295 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.876349 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:48.876921 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 21:50:48.877011 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 21:50:48.877157 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.871929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:55429 (local address 127.0.193.195:41471)
0620 21:50:48.872041 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:48.872047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:48.872172 (+ 125us) server_negotiation.cc:407] Connection header received
0620 21:50:48.872301 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:48.872304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:48.872360 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:48.872440 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:48.872719 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.873581 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.875527 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.876159 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.876295 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.876349 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:48.876921 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 21:50:48.877012 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:50:48.877246 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.871897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:39791 (local address 127.0.193.193:39189)
0620 21:50:48.872194 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.872208 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:48.872313 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:48.872580 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:48.872585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:48.872598 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:48.872685 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:48.872694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.873728 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.873732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:48.875378 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:48.875391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.876354 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.876359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:48.876545 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.876636 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 21:50:48.876855 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 21:50:48.877044 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":213,"thread_start_us":121,"threads_started":1}
I20260620 21:50:48.877228 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.871836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:41471 (local address 127.0.193.193:55429)
0620 21:50:48.872016 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.872032 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:48.872146 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:48.872454 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:48.872457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:48.872471 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:48.872617 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:48.872624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.873730 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.873732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:48.875382 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:48.875391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.876354 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.876359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:48.876542 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.876636 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 21:50:48.876855 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 21:50:48.877044 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":236}
I20260620 21:50:48.877904 7951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6d535d1a754c48bffdea62333a9945" candidate_uuid: "0df4a56890b34a72be8f690aa2009dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3738bd4c059c46f1985c0584596a312f" is_pre_election: true
I20260620 21:50:48.878185 7951 raft_consensus.cc:2454] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0df4a56890b34a72be8f690aa2009dcf in term 0.
I20260620 21:50:48.878176 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6d535d1a754c48bffdea62333a9945" candidate_uuid: "0df4a56890b34a72be8f690aa2009dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb7ce4170754c97b58212aa70323c9f" is_pre_election: true
I20260620 21:50:48.878494 8088 raft_consensus.cc:2454] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0df4a56890b34a72be8f690aa2009dcf in term 0.
I20260620 21:50:48.878651 7749 leader_election.cc:302] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df4a56890b34a72be8f690aa2009dcf, 3738bd4c059c46f1985c0584596a312f; no voters:
I20260620 21:50:48.878981 8231 raft_consensus.cc:2781] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:48.879040 8231 raft_consensus.cc:482] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:48.879067 8231 raft_consensus.cc:3037] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:48.880131 8231 raft_consensus.cc:504] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } } peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } }
I20260620 21:50:48.880570 8231 leader_election.cc:288] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [CANDIDATE]: Term 1 election: Requested vote from peers 9fb7ce4170754c97b58212aa70323c9f (127.0.193.195:41471), 3738bd4c059c46f1985c0584596a312f (127.0.193.194:39791)
I20260620 21:50:48.881045 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6d535d1a754c48bffdea62333a9945" candidate_uuid: "0df4a56890b34a72be8f690aa2009dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb7ce4170754c97b58212aa70323c9f"
I20260620 21:50:48.881067 7951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6d535d1a754c48bffdea62333a9945" candidate_uuid: "0df4a56890b34a72be8f690aa2009dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3738bd4c059c46f1985c0584596a312f"
I20260620 21:50:48.881193 7951 raft_consensus.cc:3037] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:48.881193 8088 raft_consensus.cc:3037] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:48.882375 8088 raft_consensus.cc:2454] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0df4a56890b34a72be8f690aa2009dcf in term 1.
I20260620 21:50:48.882375 7951 raft_consensus.cc:2454] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0df4a56890b34a72be8f690aa2009dcf in term 1.
I20260620 21:50:48.882872 7748 leader_election.cc:302] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0df4a56890b34a72be8f690aa2009dcf, 3738bd4c059c46f1985c0584596a312f, 9fb7ce4170754c97b58212aa70323c9f; no voters:
I20260620 21:50:48.883317 8231 raft_consensus.cc:2781] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:48.883587 8231 raft_consensus.cc:686] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [term 1 LEADER]: Becoming Leader. State: Replica: 0df4a56890b34a72be8f690aa2009dcf, State: Running, Role: LEADER
I20260620 21:50:48.884335 8231 consensus_queue.cc:237] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } } peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } }
I20260620 21:50:48.884503 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.884242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.193:40135)
0620 21:50:48.884365 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.884426 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 21:50:48.886855 7496 catalog_manager.cc:5495] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0df4a56890b34a72be8f690aa2009dcf (127.0.193.193). New cstate: current_term: 1 leader_uuid: "0df4a56890b34a72be8f690aa2009dcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0df4a56890b34a72be8f690aa2009dcf" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 } health_report { overall_health: UNKNOWN } } }
I20260620 21:50:48.905737 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.901927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.0.193.193:41503)
0620 21:50:48.902048 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:48.902052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:48.902064 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:50:48.902133 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:48.902134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:48.902202 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:48.902290 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:48.902545 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.903132 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.904100 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:48.904417 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:48.904503 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:48.904805 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:48.905324 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:48.905447 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:48.905529 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 21:50:48.905586 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:50:48.908679 7951 raft_consensus.cc:1261] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f [term 1 FOLLOWER]: Refusing update from remote peer 0df4a56890b34a72be8f690aa2009dcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:48.908857 8088 raft_consensus.cc:1261] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f [term 1 FOLLOWER]: Refusing update from remote peer 0df4a56890b34a72be8f690aa2009dcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:48.909263 8240 consensus_queue.cc:1035] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3738bd4c059c46f1985c0584596a312f" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:48.909425 8231 consensus_queue.cc:1035] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fb7ce4170754c97b58212aa70323c9f" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:48.910486 8136 heartbeater.cc:503] Master 127.0.193.253:38733 was elected leader, sending a full tablet report...
I20260620 21:50:48.911388 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.911074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.194:50053)
0620 21:50:48.911249 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.911309 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 21:50:48.912015 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.911743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.195:53027)
0620 21:50:48.911886 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.911942 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 21:50:48.915397 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.915087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.195:41647)
0620 21:50:48.915254 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.915309 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 21:50:48.915779 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.915484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.193:51167)
0620 21:50:48.915647 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.915706 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 21:50:48.916062 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.915578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.194:51785)
0620 21:50:48.915942 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.915991 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295}
I20260620 21:50:48.916816 8245 mvcc.cc:205] Tried to move back new op lower bound from 7299040251524919296 to 7299040251432763392. Current Snapshot: MvccSnapshot[applied={T|T < 7299040251524919296}]
I20260620 21:50:48.918987 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.918768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:44219 (local address 127.0.193.193:42175)
0620 21:50:48.918869 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.918926 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 21:50:48.926065 7496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:48.926275 7496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:48.932134 7496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a225ae546cb44aa8a305b263966c267f: Sending DeleteTablet for 3 replicas of tablet da6d535d1a754c48bffdea62333a9945
I20260620 21:50:48.932981 8068 tablet_service.cc:1514] Processing DeleteTablet for tablet da6d535d1a754c48bffdea62333a9945 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:48 UTC) from {username='oryx', principal='oryx/127.0.193.253@KRBTEST.COM'} at 127.0.0.1:53108
I20260620 21:50:48.932965 7931 tablet_service.cc:1514] Processing DeleteTablet for tablet da6d535d1a754c48bffdea62333a9945 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:48 UTC) from {username='oryx', principal='oryx/127.0.193.253@KRBTEST.COM'} at 127.0.0.1:50428
I20260620 21:50:48.933384 8253 tablet_replica.cc:330] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f: stopping tablet replica
I20260620 21:50:48.933490 7794 tablet_service.cc:1514] Processing DeleteTablet for tablet da6d535d1a754c48bffdea62333a9945 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:48 UTC) from {username='oryx', principal='oryx/127.0.193.253@KRBTEST.COM'} at 127.0.0.1:37878
I20260620 21:50:48.933560 8253 raft_consensus.cc:2229] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:48.933847 8253 raft_consensus.cc:2258] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:48.934152 8255 tablet_replica.cc:330] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf: stopping tablet replica
I20260620 21:50:48.934274 8255 raft_consensus.cc:2229] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:50:48.934597 8255 raft_consensus.cc:2258] T da6d535d1a754c48bffdea62333a9945 P 0df4a56890b34a72be8f690aa2009dcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:48.934859 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 7639
I20260620 21:50:48.936879 8254 tablet_replica.cc:330] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f: stopping tablet replica
I20260620 21:50:48.937047 8254 raft_consensus.cc:2229] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:48.937224 8254 raft_consensus.cc:2258] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:48.938069 8254 ts_tablet_manager.cc:1905] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:48.938019 8253 ts_tablet_manager.cc:1905] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:48.940567 8254 ts_tablet_manager.cc:1918] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:48.940672 8254 log.cc:1196] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/da6d535d1a754c48bffdea62333a9945
I20260620 21:50:48.940956 8254 ts_tablet_manager.cc:1939] T da6d535d1a754c48bffdea62333a9945 P 9fb7ce4170754c97b58212aa70323c9f: Deleting consensus metadata
I20260620 21:50:48.941839 8253 ts_tablet_manager.cc:1918] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:48.941931 8253 log.cc:1196] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/da6d535d1a754c48bffdea62333a9945
I20260620 21:50:48.942044 7481 catalog_manager.cc:4841] TS 9fb7ce4170754c97b58212aa70323c9f (127.0.193.195:41471): tablet da6d535d1a754c48bffdea62333a9945 (table test-table [id=110aff8fe26f4d6695b6ccb1ad529034]) successfully deleted
I20260620 21:50:48.942232 8253 ts_tablet_manager.cc:1939] T da6d535d1a754c48bffdea62333a9945 P 3738bd4c059c46f1985c0584596a312f: Deleting consensus metadata
I20260620 21:50:48.943212 7482 catalog_manager.cc:4841] TS 3738bd4c059c46f1985c0584596a312f (127.0.193.194:39791): tablet da6d535d1a754c48bffdea62333a9945 (table test-table [id=110aff8fe26f4d6695b6ccb1ad529034]) successfully deleted
I20260620 21:50:48.944203 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 7868
I20260620 21:50:48.944262 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:48.944002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:41503 (local address 127.0.0.1:37898)
0620 21:50:48.944118 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:48.944177 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 21:50:48.945367 7483 catalog_manager.cc:4584] TS 0df4a56890b34a72be8f690aa2009dcf (127.0.193.193:41503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da6d535d1a754c48bffdea62333a9945: Network error: Client connection negotiation failed: client connection to 127.0.193.193:41503: connect: Connection refused (error 111)
I20260620 21:50:48.951143 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 8004
I20260620 21:50:48.957973 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 7359
I20260620 21:50:48.964298 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 7461
2026-06-20T21:50:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5575 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:50:49 dist-test-slave-kk5c krb5kdc[8258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:50:49 dist-test-slave-kk5c krb5kdc[8258](info): set up 2 sockets
Jun 20 21:50:49 dist-test-slave-kk5c krb5kdc[8258](info): commencing operation
krb5kdc: starting...
W20260620 21:50:50.031855 775 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:50:50 dist-test-slave-kk5c krb5kdc[8258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.254.keytab.
Entry for principal oryx/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.254.keytab.
I20260620 21:50:50.162751 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:33053
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.254.keytab
--principal=oryx/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:33053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:50.278228 8271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:50.278635 8271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:50.278714 8271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:50.283382 8271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:50.283517 8271 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 21:50:50.283550 8271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:50.283591 8271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:50.283653 8271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:50.283692 8271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:50.289424 8271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:33053
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.254.keytab
--principal=oryx/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:33053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:50.290576 8271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:50.291642 8271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:50.300596 8279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:50.300624 8277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:50.300596 8276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:50.300647 8271 server_base.cc:1034] running on GCE node
I20260620 21:50:50.301512 8271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:50.302690 8271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:50.303887 8271 hybrid_clock.cc:648] HybridClock initialized: now 1781992250303867 us; error 35 us; skew 500 ppm
Jun 20 21:50:50 dist-test-slave-kk5c krb5kdc[8258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:50.311028 8271 init.cc:377] Logged in from keytab as oryx/127.0.193.254@KRBTEST.COM (short username oryx)
I20260620 21:50:50.312166 8271 webserver.cc:460] Webserver started at http://127.0.193.254:33481/ using document root <none> and password file <none>
I20260620 21:50:50.312819 8271 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:50.312922 8271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:50.313148 8271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:50.315589 8271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "92b57d2ea2724612bfe812312cf465db"
format_stamp: "Formatted at 2026-06-20 21:50:50 on dist-test-slave-kk5c"
server_key: "f455236ea19b559db84966f119898944"
server_key_iv: "c229069287024055f46ae05252852b17"
server_key_version: "encryptionkey@0"
I20260620 21:50:50.316217 8271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "92b57d2ea2724612bfe812312cf465db"
format_stamp: "Formatted at 2026-06-20 21:50:50 on dist-test-slave-kk5c"
server_key: "f455236ea19b559db84966f119898944"
server_key_iv: "c229069287024055f46ae05252852b17"
server_key_version: "encryptionkey@0"
I20260620 21:50:50.320446 8271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:50:50.323374 8286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:50.324864 8271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:50.324980 8271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "92b57d2ea2724612bfe812312cf465db"
format_stamp: "Formatted at 2026-06-20 21:50:50 on dist-test-slave-kk5c"
server_key: "f455236ea19b559db84966f119898944"
server_key_iv: "c229069287024055f46ae05252852b17"
server_key_version: "encryptionkey@0"
I20260620 21:50:50.325141 8271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:50.354092 8271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:50.357472 8271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:50.357681 8271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:50.366168 8271 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:33053
I20260620 21:50:50.366333 8348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:33053 every 8 connection(s)
I20260620 21:50:50.367154 8271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:50.369961 8349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:50.370611 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 8271
I20260620 21:50:50.370790 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:50.371090 775 external_mini_cluster.cc:1428] Setting key de7f09448bb17fb792634cdb33a3a36e
I20260620 21:50:50.376329 8349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db: Bootstrap starting.
I20260620 21:50:50.379096 8349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:50.380019 8349 log.cc:824] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:50:50 dist-test-slave-kk5c krb5kdc[8258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
I20260620 21:50:50.382620 8349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db: No bootstrap required, opened a new log
I20260620 21:50:50.386700 8349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b57d2ea2724612bfe812312cf465db" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 33053 } }
I20260620 21:50:50.386988 8349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:50.387035 8349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b57d2ea2724612bfe812312cf465db, State: Initialized, Role: FOLLOWER
I20260620 21:50:50.387276 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:50.372455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.0.193.254:33053)
0620 21:50:50.372876 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:50.372891 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:50.372926 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:50:50.373550 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:50.373566 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:50.373896 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:50.374185 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:50.375094 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.376173 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.377303 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.377765 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.377880 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:50.380780 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:50.380785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:50.380800 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:50.380824 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:50.383619 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:50.384235 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:50.384241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:50.384248 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:50.384395 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:50.384908 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:50.384911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:50.384913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:50.385453 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:50.385657 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:50.385930 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 21:50:50.386251 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":139,"threads_started":1}
I20260620 21:50:50.387552 8349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b57d2ea2724612bfe812312cf465db" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 33053 } }
I20260620 21:50:50.387708 8349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:50.387774 8349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:50.387867 8349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:50.389075 8349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b57d2ea2724612bfe812312cf465db" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 33053 } }
I20260620 21:50:50.389444 8349 leader_election.cc:302] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92b57d2ea2724612bfe812312cf465db; no voters:
I20260620 21:50:50.389777 8349 leader_election.cc:288] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:50.389972 8354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:50.390189 8354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [term 1 LEADER]: Becoming Leader. State: Replica: 92b57d2ea2724612bfe812312cf465db, State: Running, Role: LEADER
I20260620 21:50:50.390537 8354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b57d2ea2724612bfe812312cf465db" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 33053 } }
I20260620 21:50:50.391556 8349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:50.392309 8356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92b57d2ea2724612bfe812312cf465db. Latest consensus state: current_term: 1 leader_uuid: "92b57d2ea2724612bfe812312cf465db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b57d2ea2724612bfe812312cf465db" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 33053 } } }
I20260620 21:50:50.392477 8356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:50.392787 8355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92b57d2ea2724612bfe812312cf465db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b57d2ea2724612bfe812312cf465db" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 33053 } } }
I20260620 21:50:50.392912 8355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:50.393527 8361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:50.397343 8361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:50.403532 8361 catalog_manager.cc:1347] Generated new cluster ID: 0b9167d6fd7944abaa1139046787015d
I20260620 21:50:50.403620 8361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:50.415638 8361 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:50.416704 8361 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:50.421938 8361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db: Generated new TSK 0
I20260620 21:50:50.422752 8361 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.193.keytab.
Entry for principal oryx/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.193.keytab.
I20260620 21:50:50.455524 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:33053
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.193.keytab
--principal=oryx/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:50.573262 8375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:50.573633 8375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:50.573705 8375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:50.578540 8375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:50.578660 8375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:50.578735 8375 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:50:50.578801 8375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:50.584951 8375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.193.keytab
--principal=oryx/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.193.254:33053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:50.586122 8375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:50.587276 8375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:50.598039 8380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:50.598042 8381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:50.598039 8383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:50.598392 8375 server_base.cc:1034] running on GCE node
I20260620 21:50:50.598959 8375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:50.599732 8375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:50.600950 8375 hybrid_clock.cc:648] HybridClock initialized: now 1781992250600929 us; error 33 us; skew 500 ppm
Jun 20 21:50:50 dist-test-slave-kk5c krb5kdc[8258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:50.604269 8375 init.cc:377] Logged in from keytab as oryx/127.0.193.193@KRBTEST.COM (short username oryx)
I20260620 21:50:50.605235 8375 webserver.cc:460] Webserver started at http://127.0.193.193:38863/ using document root <none> and password file <none>
I20260620 21:50:50.605800 8375 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:50.605866 8375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:50.606053 8375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:50.608458 8375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "969bc97c9ce149a79d2697b759b4c45a"
format_stamp: "Formatted at 2026-06-20 21:50:50 on dist-test-slave-kk5c"
server_key: "0c59da7aae624ad02fb41e0bab16a5f9"
server_key_iv: "9e8c9fa393ae682f1dd1dc86f519ac2e"
server_key_version: "encryptionkey@0"
I20260620 21:50:50.609032 8375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "969bc97c9ce149a79d2697b759b4c45a"
format_stamp: "Formatted at 2026-06-20 21:50:50 on dist-test-slave-kk5c"
server_key: "0c59da7aae624ad02fb41e0bab16a5f9"
server_key_iv: "9e8c9fa393ae682f1dd1dc86f519ac2e"
server_key_version: "encryptionkey@0"
I20260620 21:50:50.613355 8375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 21:50:50.616669 8390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:50.618165 8375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:50:50.618388 8375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "969bc97c9ce149a79d2697b759b4c45a"
format_stamp: "Formatted at 2026-06-20 21:50:50 on dist-test-slave-kk5c"
server_key: "0c59da7aae624ad02fb41e0bab16a5f9"
server_key_iv: "9e8c9fa393ae682f1dd1dc86f519ac2e"
server_key_version: "encryptionkey@0"
I20260620 21:50:50.618628 8375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:50.640590 8375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:50.644902 8375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:50.645160 8375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:50.646816 8375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:50.646898 8375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:50.646972 8375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:50.647020 8375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:50:50 dist-test-slave-kk5c krb5kdc[8258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
I20260620 21:50:50.659711 8375 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:46347
I20260620 21:50:50.659716 8505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:46347 every 8 connection(s)
I20260620 21:50:50.660970 8375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:50.663493 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:50.647745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:54829 (local address 127.0.193.254:33053)
0620 21:50:50.647919 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:50.647926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:50.648795 (+ 869us) server_negotiation.cc:407] Connection header received
0620 21:50:50.649947 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:50.649952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:50.650028 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:50.650128 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:50.651908 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.652823 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.654651 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.655036 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.655230 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:50.658938 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:50.658942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:50.658947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:50.658983 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:50.660799 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:50.661528 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:50.661532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:50.661534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:50.661635 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:50.662036 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:50.662039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:50.662041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:50.662295 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:50.662394 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:50.663153 (+ 759us) server_negotiation.cc:299] Negotiation successful
0620 21:50:50.663366 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 21:50:50.663957 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 8375
I20260620 21:50:50.664105 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:50:50.664412 775 external_mini_cluster.cc:1428] Setting key 2673f050844860fa059e3421813c8fd3
I20260620 21:50:50.664698 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:50.648140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:33053 (local address 127.0.193.193:54829)
0620 21:50:50.648602 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:50.648658 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:50.649671 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:50.650534 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:50.650541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:50.651012 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:50.651705 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:50.651744 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.653010 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.653020 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:50.654397 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:50.654409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.655343 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.655351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:50.655596 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:50.656379 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:50.656407 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:50.658711 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:50.660955 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:50.660960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:50.660973 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:50.661370 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:50.661737 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:50.661740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:50.661742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:50.661915 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:50.662455 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:50.662460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:50.662711 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 21:50:50.662979 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 21:50:50.663530 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":139,"threads_started":1}
I20260620 21:50:50.666584 8506 heartbeater.cc:347] Connected to a master server at 127.0.193.254:33053
I20260620 21:50:50.666857 8506 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:50.667528 8506 heartbeater.cc:511] Master 127.0.193.254:33053 requested a full tablet report, sending...
I20260620 21:50:50.669459 8302 ts_manager.cc:194] Registered new tserver with Master: 969bc97c9ce149a79d2697b759b4c45a (127.0.193.193:46347)
I20260620 21:50:50.671207 8302 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:54829
WARNING: no policy specified for oryx/127.0.193.194@KRBTEST.COM; defaulting to no policy
I20260620 21:50:50.678236 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:50.669239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:33053 (local address 127.0.193.193:57823)
0620 21:50:50.669605 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:50.669623 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:50.669762 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:50.670024 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:50.670026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:50.670250 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:50.670350 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:50.670356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.671425 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.671431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:50.672216 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:50.672226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.672937 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.672941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:50.673052 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:50.673695 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:50.673712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:50.674208 (+ 496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:50.676357 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:50.676360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:50.676361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:50.676582 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:50.676930 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:50.676933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:50.676934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:50.677065 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:50.677708 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:50.677711 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:50.677798 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 21:50:50.677937 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 21:50:50.678071 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":115,"threads_started":1}
I20260620 21:50:50.678412 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:50.669300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:57823 (local address 127.0.193.254:33053)
0620 21:50:50.669424 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:50.669428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:50.669687 (+ 259us) server_negotiation.cc:407] Connection header received
0620 21:50:50.669864 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:50.669867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:50.669917 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:50.670007 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:50.670472 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.671289 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.672347 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.672781 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.672915 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:50.674373 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:50.674379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:50.674381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:50.674415 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:50.676246 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:50.676728 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:50.676731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:50.676733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:50.676813 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:50.677343 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:50.677346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:50.677347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:50.677533 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:50.677624 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:50.678091 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 21:50:50.678203 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.194.keytab.
Entry for principal oryx/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.194.keytab.
I20260620 21:50:50.696292 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:33053
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.194.keytab
--principal=oryx/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:50.812283 8515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:50.812573 8515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:50.812620 8515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:50.816763 8515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:50.816835 8515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:50.816877 8515 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:50:50.816910 8515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:50.822707 8515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.194.keytab
--principal=oryx/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.193.254:33053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:50.823880 8515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:50.824965 8515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:50.836081 8523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:50.836066 8521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:50.836164 8515 server_base.cc:1034] running on GCE node
W20260620 21:50:50.836071 8520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:50.837255 8515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:50.837922 8515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:50.839141 8515 hybrid_clock.cc:648] HybridClock initialized: now 1781992250839120 us; error 40 us; skew 500 ppm
Jun 20 21:50:50 dist-test-slave-kk5c krb5kdc[8258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:50.842507 8515 init.cc:377] Logged in from keytab as oryx/127.0.193.194@KRBTEST.COM (short username oryx)
I20260620 21:50:50.843566 8515 webserver.cc:460] Webserver started at http://127.0.193.194:44507/ using document root <none> and password file <none>
I20260620 21:50:50.844101 8515 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:50.844169 8515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:50.844342 8515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:50.846755 8515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "10b199fe0bfd4600afdd04c76a0583ee"
format_stamp: "Formatted at 2026-06-20 21:50:50 on dist-test-slave-kk5c"
server_key: "48a561c8c5534a40fb9068addf7f996d"
server_key_iv: "a277427254ebf8c723d4ffe4875bf9db"
server_key_version: "encryptionkey@0"
I20260620 21:50:50.847427 8515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "10b199fe0bfd4600afdd04c76a0583ee"
format_stamp: "Formatted at 2026-06-20 21:50:50 on dist-test-slave-kk5c"
server_key: "48a561c8c5534a40fb9068addf7f996d"
server_key_iv: "a277427254ebf8c723d4ffe4875bf9db"
server_key_version: "encryptionkey@0"
I20260620 21:50:50.851569 8515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:50:50.854389 8530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:50.856232 8515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:50.856354 8515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "10b199fe0bfd4600afdd04c76a0583ee"
format_stamp: "Formatted at 2026-06-20 21:50:50 on dist-test-slave-kk5c"
server_key: "48a561c8c5534a40fb9068addf7f996d"
server_key_iv: "a277427254ebf8c723d4ffe4875bf9db"
server_key_version: "encryptionkey@0"
I20260620 21:50:50.856460 8515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:50.873826 8515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:50.877408 8515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:50.877594 8515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:50.879060 8515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:50.879115 8515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:50.879165 8515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:50.879248 8515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:50:50 dist-test-slave-kk5c krb5kdc[8258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
I20260620 21:50:50.890573 8515 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:43733
I20260620 21:50:50.890895 8645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:43733 every 8 connection(s)
I20260620 21:50:50.891728 8515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:50.893605 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 8515
I20260620 21:50:50.893728 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:50:50.894042 775 external_mini_cluster.cc:1428] Setting key 628f4be2ef79606ad1ba4287f555b347
I20260620 21:50:50.894275 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:50.880245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:56117 (local address 127.0.193.254:33053)
0620 21:50:50.880407 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:50.880415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:50.881369 (+ 954us) server_negotiation.cc:407] Connection header received
0620 21:50:50.882889 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:50.882897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:50.882962 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:50.883068 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:50.884530 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.885056 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.886264 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.886627 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.886747 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:50.889989 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:50.889992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:50.889994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:50.890020 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:50.891636 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:50.892490 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:50.892497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:50.892500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:50.892607 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:50.892986 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:50.892989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:50.892991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:50.893204 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:50.893318 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:50.893961 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 21:50:50.894107 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:50:50.895244 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:50.880661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:33053 (local address 127.0.193.194:56117)
0620 21:50:50.881182 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:50.881244 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:50.882504 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:50.883303 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:50.883310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:50.883767 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:50.884370 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:50.884384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.885199 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.885203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:50.886076 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:50.886085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.886865 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.886874 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:50.887013 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:50.887992 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:50.888010 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:50.889803 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:50.891784 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:50.891791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:50.891806 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:50.892158 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:50.892728 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:50.892732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:50.892734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:50.892875 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:50.893315 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:50.893322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:50.893636 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 21:50:50.893876 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 21:50:50.894234 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":142,"threads_started":1}
I20260620 21:50:50.896766 8646 heartbeater.cc:347] Connected to a master server at 127.0.193.254:33053
I20260620 21:50:50.896919 8646 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:50.897495 8646 heartbeater.cc:511] Master 127.0.193.254:33053 requested a full tablet report, sending...
I20260620 21:50:50.898557 8303 ts_manager.cc:194] Registered new tserver with Master: 10b199fe0bfd4600afdd04c76a0583ee (127.0.193.194:43733)
I20260620 21:50:50.899412 8303 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:56117
WARNING: no policy specified for oryx/127.0.193.195@KRBTEST.COM; defaulting to no policy
I20260620 21:50:50.907569 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:50.899059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:36669 (local address 127.0.193.254:33053)
0620 21:50:50.899221 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:50.899225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:50.899456 (+ 231us) server_negotiation.cc:407] Connection header received
0620 21:50:50.899602 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:50.899608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:50.899663 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:50.899755 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:50.900457 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.901201 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.902568 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.902793 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.902882 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:50.904217 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:50.904220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:50.904222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:50.904247 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:50.905697 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:50.906173 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:50.906177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:50.906178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:50.906260 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:50.906585 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:50.906590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:50.906591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:50.906808 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:50.906978 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:50.907269 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 21:50:50.907408 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:50:50.907583 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:50.898972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:33053 (local address 127.0.193.194:36669)
0620 21:50:50.899360 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:50.899375 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:50.899477 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:50.899884 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:50.899890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:50.900133 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:50.900324 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:50.900335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.901341 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.901345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:50.902431 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:50.902445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:50.902920 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:50.902924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:50.903031 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:50.903598 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:50.903613 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:50.904073 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:50.905834 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:50.905837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:50.905838 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:50.906064 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:50.906354 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:50.906359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:50.906361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:50.906474 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:50.906968 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:50.906972 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:50.907074 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 21:50:50.907223 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 21:50:50.907380 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.195.keytab.
Entry for principal oryx/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.195.keytab.
I20260620 21:50:50.929563 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:33053
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.195.keytab
--principal=oryx/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:51.056860 8655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:51.057188 8655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:51.057278 8655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:51.061718 8655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:51.061822 8655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:51.061895 8655 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 21:50:51.061982 8655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:51.067790 8655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/krb5kdc/oryx_127.0.193.195.keytab
--principal=oryx/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.193.254:33053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:51.068966 8655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:51.070014 8655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:51.081061 8661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:51.081084 8660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:51.081509 8663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:51.081581 8655 server_base.cc:1034] running on GCE node
I20260620 21:50:51.082090 8655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:51.082801 8655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:51.084002 8655 hybrid_clock.cc:648] HybridClock initialized: now 1781992251083944 us; error 72 us; skew 500 ppm
Jun 20 21:50:51 dist-test-slave-kk5c krb5kdc[8258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:51.087795 8655 init.cc:377] Logged in from keytab as oryx/127.0.193.195@KRBTEST.COM (short username oryx)
I20260620 21:50:51.089097 8655 webserver.cc:460] Webserver started at http://127.0.193.195:39287/ using document root <none> and password file <none>
I20260620 21:50:51.089717 8655 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:51.089815 8655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:51.090060 8655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:51.092453 8655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "6c47a35e5e3942b194046a14d57441df"
format_stamp: "Formatted at 2026-06-20 21:50:51 on dist-test-slave-kk5c"
server_key: "5ef1c1238c39df5b2d0137e8bb9550f7"
server_key_iv: "77d55d93d0a89b6d36f4f2376577b849"
server_key_version: "encryptionkey@0"
I20260620 21:50:51.093184 8655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "6c47a35e5e3942b194046a14d57441df"
format_stamp: "Formatted at 2026-06-20 21:50:51 on dist-test-slave-kk5c"
server_key: "5ef1c1238c39df5b2d0137e8bb9550f7"
server_key_iv: "77d55d93d0a89b6d36f4f2376577b849"
server_key_version: "encryptionkey@0"
I20260620 21:50:51.098066 8655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 21:50:51.101212 8670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:51.102723 8655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:50:51.102902 8655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "6c47a35e5e3942b194046a14d57441df"
format_stamp: "Formatted at 2026-06-20 21:50:51 on dist-test-slave-kk5c"
server_key: "5ef1c1238c39df5b2d0137e8bb9550f7"
server_key_iv: "77d55d93d0a89b6d36f4f2376577b849"
server_key_version: "encryptionkey@0"
I20260620 21:50:51.103055 8655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:51.116916 8655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:51.121125 8655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:51.121367 8655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:51.123413 8655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:51.123502 8655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:51.123566 8655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:51.123603 8655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 21:50:51 dist-test-slave-kk5c krb5kdc[8258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992251, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
I20260620 21:50:51.135051 8655 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:41167
I20260620 21:50:51.135180 8785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:41167 every 8 connection(s)
I20260620 21:50:51.136001 8655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:50:51.137778 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 8655
I20260620 21:50:51.137936 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:50:51.138319 775 external_mini_cluster.cc:1428] Setting key 74dbeb09a613f571072b1dc291bf7add
I20260620 21:50:51.139298 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.124113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:37435 (local address 127.0.193.254:33053)
0620 21:50:51.124267 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.124271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.125515 (+ 1244us) server_negotiation.cc:407] Connection header received
0620 21:50:51.126890 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.126895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.126959 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.127007 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:51.128872 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.129804 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.131539 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.131923 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.132049 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.135050 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:51.135054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:51.135057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:51.135092 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:51.136956 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:51.137446 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:51.137449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:51.137451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:51.137534 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:51.137964 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:51.137968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:51.137969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:51.138240 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:51.138321 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:51.138897 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 21:50:51.139027 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:50:51.140300 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.124686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:33053 (local address 127.0.193.195:37435)
0620 21:50:51.125345 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:51.125393 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:51.126565 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:51.127484 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:51.127496 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:51.127976 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:51.128713 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.128729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.129977 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.129981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.131404 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.131417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.132202 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.132205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.132359 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.133129 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:51.133155 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:51.134812 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:51.137062 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:51.137068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:51.137081 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:51.137333 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:51.137655 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:51.137663 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:51.137666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:51.137821 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:51.138355 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:51.138362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:51.138678 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 21:50:51.138847 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 21:50:51.139181 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":174,"threads_started":1}
I20260620 21:50:51.141463 8786 heartbeater.cc:347] Connected to a master server at 127.0.193.254:33053
I20260620 21:50:51.141600 8786 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:51.142163 8786 heartbeater.cc:511] Master 127.0.193.254:33053 requested a full tablet report, sending...
I20260620 21:50:51.143334 8302 ts_manager.cc:194] Registered new tserver with Master: 6c47a35e5e3942b194046a14d57441df (127.0.193.195:41167)
I20260620 21:50:51.144057 8302 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:37435
I20260620 21:50:51.151743 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.143971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:34713 (local address 127.0.193.254:33053)
0620 21:50:51.144103 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.144107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.144338 (+ 231us) server_negotiation.cc:407] Connection header received
0620 21:50:51.144526 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.144528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.144566 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.144623 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:51.145129 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.145858 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.146811 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.147077 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.147175 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.148317 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:51.148320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:51.148322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:51.148343 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:51.149912 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:51.150336 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:51.150339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:51.150340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:51.150413 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:51.150723 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:51.150726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:51.150728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:51.150949 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:51.151029 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:51.151452 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 21:50:51.151576 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:50:51.151772 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.143903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:33053 (local address 127.0.193.195:34713)
0620 21:50:51.144240 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:51.144257 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:51.144398 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:51.144667 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:51.144673 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:51.144899 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:51.145012 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.145018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.145998 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.146001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.146619 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.146628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.147247 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.147255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.147326 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.147850 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:51.147861 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:51.148182 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:51.150053 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:51.150056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:51.150057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:51.150247 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:51.150520 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:51.150522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:51.150524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:51.150608 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:51.151100 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:51.151103 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:51.151258 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 21:50:51.151393 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 21:50:51.151598 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":109,"threads_started":1}
I20260620 21:50:51.152439 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 21:50:51 dist-test-slave-kk5c krb5kdc[8258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992251, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 21:50:51 dist-test-slave-kk5c krb5kdc[8258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:51 dist-test-slave-kk5c krb5kdc[8258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 21:50:51.179384 8352 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:51.172402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.0.193.254:33053)
0620 21:50:51.172531 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.172535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.172548 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:50:51.172590 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.172593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.172631 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.172723 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:51.173658 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.174209 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.175267 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.175497 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.175586 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.179243 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37448: BlockingRecv error: recv got EOF from 127.0.0.1:37448 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 21:50:51 dist-test-slave-kk5c krb5kdc[8258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992251, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
I20260620 21:50:51.190133 8352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.181106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.0.193.254:33053)
0620 21:50:51.181276 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.181281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.181406 (+ 125us) server_negotiation.cc:407] Connection header received
0620 21:50:51.181565 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.181568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.181617 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.181714 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:51.182364 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.182883 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.183760 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.183976 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.184089 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.186657 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:51.186660 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:51.186662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:51.186703 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:51.188280 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:51.188827 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:51.188831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:51.188832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:51.188923 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:51.189206 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:51.189208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:51.189210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:51.189395 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:51.189511 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:51.189809 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 21:50:51.189908 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 21:50:51.193343 8302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:50:51.196565 8302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:51.211915 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.206842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.0.193.193:46347)
0620 21:50:51.207558 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.207565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.207589 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:50:51.207660 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.207665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.207834 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.207975 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:51.208219 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.208923 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.210587 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.211267 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.211387 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.211538 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:51.211671 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 21:50:51.211768 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":91,"threads_started":1}
I20260620 21:50:51.211915 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.206416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:46347 (local address 127.0.0.1:56910)
0620 21:50:51.206846 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:51.206859 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:51.206934 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:51.207966 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:51.207969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:51.207985 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:51.208089 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.208098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.209037 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.209043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.210455 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.210465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.211385 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.211389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.211489 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.211524 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 21:50:51.211642 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 21:50:51.211767 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":29,"thread_start_us":102,"threads_started":1}
I20260620 21:50:51.212139 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.205708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:41167 (local address 127.0.0.1:47326)
0620 21:50:51.206562 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:51.206623 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:51.206817 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:51.207644 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:51.207650 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:51.207686 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:51.207979 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.207989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.209397 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.209400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.210649 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.210656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.211728 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.211732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.211828 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.211849 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:50:51.211919 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 21:50:51.211984 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":181,"spinlock_wait_cycles":5888,"thread_start_us":94,"threads_started":1}
I20260620 21:50:51.212626 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.205897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.0.193.195:41167)
0620 21:50:51.206479 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.206495 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.206721 (+ 226us) server_negotiation.cc:407] Connection header received
0620 21:50:51.207018 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.207024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.207294 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.207449 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:51.208099 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.209272 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.210762 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.211410 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.211510 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.211666 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:51.212376 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 21:50:51.212472 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":124,"threads_started":1}
I20260620 21:50:51.212821 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.206117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:43733 (local address 127.0.0.1:43778)
0620 21:50:51.206634 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:51.206650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:51.206818 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:51.207644 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:51.207650 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:51.207694 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:51.207915 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.207923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.209409 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.209412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.210633 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.210642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.212433 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.212437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.212519 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.212540 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:50:51.212619 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 21:50:51.212694 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":121,"spinlock_wait_cycles":6144,"thread_start_us":87,"threads_started":1}
I20260620 21:50:51.213275 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.206486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.0.193.194:43733)
0620 21:50:51.207151 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.207162 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.207212 (+ 50us) server_negotiation.cc:407] Connection header received
0620 21:50:51.207289 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.207294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.207484 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.207639 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:51.208060 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.209272 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.211295 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.212088 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.212208 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.212376 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:51.212670 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 21:50:51.213128 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":110,"threads_started":1}
I20260620 21:50:51.215579 8580 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3ba5b6f86441229b4b7a4cc03d694a (DEFAULT_TABLE table=test-table [id=d81f44ddff52483faf1777f9caf5a280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:51.215652 8720 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3ba5b6f86441229b4b7a4cc03d694a (DEFAULT_TABLE table=test-table [id=d81f44ddff52483faf1777f9caf5a280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:51.215612 8440 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3ba5b6f86441229b4b7a4cc03d694a (DEFAULT_TABLE table=test-table [id=d81f44ddff52483faf1777f9caf5a280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:51.216620 8580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3ba5b6f86441229b4b7a4cc03d694a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:51.216629 8720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3ba5b6f86441229b4b7a4cc03d694a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:51.216622 8440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3ba5b6f86441229b4b7a4cc03d694a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:51.224018 8814 tablet_bootstrap.cc:492] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee: Bootstrap starting.
I20260620 21:50:51.224864 8815 tablet_bootstrap.cc:492] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a: Bootstrap starting.
I20260620 21:50:51.225042 8816 tablet_bootstrap.cc:492] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df: Bootstrap starting.
I20260620 21:50:51.226909 8814 tablet_bootstrap.cc:654] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:51.227452 8816 tablet_bootstrap.cc:654] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:51.227452 8815 tablet_bootstrap.cc:654] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:51.228368 8815 log.cc:824] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:51.228367 8816 log.cc:824] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:51.228963 8814 log.cc:824] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:51.230543 8816 tablet_bootstrap.cc:492] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df: No bootstrap required, opened a new log
I20260620 21:50:51.230543 8815 tablet_bootstrap.cc:492] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a: No bootstrap required, opened a new log
I20260620 21:50:51.230755 8815 ts_tablet_manager.cc:1397] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:50:51.230754 8816 ts_tablet_manager.cc:1397] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:50:51.231077 8814 tablet_bootstrap.cc:492] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee: No bootstrap required, opened a new log
I20260620 21:50:51.231341 8814 ts_tablet_manager.cc:1397] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 21:50:51.235119 8816 raft_consensus.cc:348] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } }
I20260620 21:50:51.235570 8816 raft_consensus.cc:374] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:51.235627 8816 raft_consensus.cc:729] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c47a35e5e3942b194046a14d57441df, State: Initialized, Role: FOLLOWER
I20260620 21:50:51.235445 8814 raft_consensus.cc:348] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } }
I20260620 21:50:51.235512 8815 raft_consensus.cc:348] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } }
I20260620 21:50:51.235713 8814 raft_consensus.cc:374] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:51.235764 8814 raft_consensus.cc:729] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b199fe0bfd4600afdd04c76a0583ee, State: Initialized, Role: FOLLOWER
I20260620 21:50:51.235765 8815 raft_consensus.cc:374] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:51.235809 8815 raft_consensus.cc:729] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969bc97c9ce149a79d2697b759b4c45a, State: Initialized, Role: FOLLOWER
I20260620 21:50:51.236166 8816 consensus_queue.cc:260] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } }
I20260620 21:50:51.236169 8815 consensus_queue.cc:260] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } }
I20260620 21:50:51.236210 8814 consensus_queue.cc:260] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } }
I20260620 21:50:51.236869 8506 heartbeater.cc:503] Master 127.0.193.254:33053 was elected leader, sending a full tablet report...
I20260620 21:50:51.237157 8786 heartbeater.cc:503] Master 127.0.193.254:33053 was elected leader, sending a full tablet report...
I20260620 21:50:51.237277 8814 ts_tablet_manager.cc:1428] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:50:51.237462 8646 heartbeater.cc:503] Master 127.0.193.254:33053 was elected leader, sending a full tablet report...
I20260620 21:50:51.238312 8816 ts_tablet_manager.cc:1428] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260620 21:50:51.238722 8815 ts_tablet_manager.cc:1428] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260620 21:50:51.271392 8821 raft_consensus.cc:482] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:51.271697 8821 raft_consensus.cc:504] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } }
I20260620 21:50:51.272990 8821 leader_election.cc:288] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 969bc97c9ce149a79d2697b759b4c45a (127.0.193.193:46347), 10b199fe0bfd4600afdd04c76a0583ee (127.0.193.194:43733)
I20260620 21:50:51.277072 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.273310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:37553 (local address 127.0.193.193:46347)
0620 21:50:51.273454 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.273459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.273591 (+ 132us) server_negotiation.cc:407] Connection header received
0620 21:50:51.273687 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.273690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.273745 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.273841 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:51.274133 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.274704 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.275951 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.276419 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.276578 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.276649 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:51.276850 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 21:50:51.276932 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:50:51.277072 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.273232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:46347 (local address 127.0.193.195:37553)
0620 21:50:51.273447 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:51.273465 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:51.273582 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:51.273853 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:51.273856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:51.273874 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:51.273999 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.274004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.274833 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.274838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.275807 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.275817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.276570 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.276574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.276709 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.276733 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:50:51.276833 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 21:50:51.276919 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 21:50:51.277588 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.273472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:59531 (local address 127.0.193.194:43733)
0620 21:50:51.273588 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.273592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.273812 (+ 220us) server_negotiation.cc:407] Connection header received
0620 21:50:51.273982 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.273984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.274024 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.274104 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:51.274789 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.275379 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.276581 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.276992 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.277130 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.277189 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:51.277367 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 21:50:51.277463 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:50:51.277935 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.273244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:43733 (local address 127.0.193.195:59531)
0620 21:50:51.273712 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:51.273725 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:51.273848 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:51.274125 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:51.274128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:51.274144 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:51.274652 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.274661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.275485 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.275492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.276470 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.276480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.277129 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.277132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.277229 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.277255 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:50:51.277752 (+ 497us) client_negotiation.cc:240] Negotiation successful
0620 21:50:51.277825 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":14,"thread_start_us":157,"threads_started":1}
I20260620 21:50:51.277897 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3ba5b6f86441229b4b7a4cc03d694a" candidate_uuid: "6c47a35e5e3942b194046a14d57441df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969bc97c9ce149a79d2697b759b4c45a" is_pre_election: true
I20260620 21:50:51.278272 8460 raft_consensus.cc:2454] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c47a35e5e3942b194046a14d57441df in term 0.
I20260620 21:50:51.278543 8600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3ba5b6f86441229b4b7a4cc03d694a" candidate_uuid: "6c47a35e5e3942b194046a14d57441df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b199fe0bfd4600afdd04c76a0583ee" is_pre_election: true
I20260620 21:50:51.278816 8600 raft_consensus.cc:2454] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c47a35e5e3942b194046a14d57441df in term 0.
I20260620 21:50:51.278833 8673 leader_election.cc:302] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c47a35e5e3942b194046a14d57441df, 969bc97c9ce149a79d2697b759b4c45a; no voters:
I20260620 21:50:51.279136 8821 raft_consensus.cc:2781] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:51.279255 8821 raft_consensus.cc:482] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:51.279320 8821 raft_consensus.cc:3037] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:51.280483 8821 raft_consensus.cc:504] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } }
I20260620 21:50:51.280942 8821 leader_election.cc:288] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [CANDIDATE]: Term 1 election: Requested vote from peers 969bc97c9ce149a79d2697b759b4c45a (127.0.193.193:46347), 10b199fe0bfd4600afdd04c76a0583ee (127.0.193.194:43733)
I20260620 21:50:51.281406 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3ba5b6f86441229b4b7a4cc03d694a" candidate_uuid: "6c47a35e5e3942b194046a14d57441df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969bc97c9ce149a79d2697b759b4c45a"
I20260620 21:50:51.281548 8460 raft_consensus.cc:3037] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:51.281570 8600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3ba5b6f86441229b4b7a4cc03d694a" candidate_uuid: "6c47a35e5e3942b194046a14d57441df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b199fe0bfd4600afdd04c76a0583ee"
I20260620 21:50:51.281737 8600 raft_consensus.cc:3037] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:51.282713 8460 raft_consensus.cc:2454] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c47a35e5e3942b194046a14d57441df in term 1.
I20260620 21:50:51.282863 8600 raft_consensus.cc:2454] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c47a35e5e3942b194046a14d57441df in term 1.
I20260620 21:50:51.283123 8673 leader_election.cc:302] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c47a35e5e3942b194046a14d57441df, 969bc97c9ce149a79d2697b759b4c45a; no voters:
I20260620 21:50:51.283334 8821 raft_consensus.cc:2781] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:51.283560 8821 raft_consensus.cc:686] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [term 1 LEADER]: Becoming Leader. State: Replica: 6c47a35e5e3942b194046a14d57441df, State: Running, Role: LEADER
I20260620 21:50:51.283972 8821 consensus_queue.cc:237] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } }
I20260620 21:50:51.287289 8303 catalog_manager.cc:5495] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c47a35e5e3942b194046a14d57441df (127.0.193.195). New cstate: current_term: 1 leader_uuid: "6c47a35e5e3942b194046a14d57441df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } health_report { overall_health: UNKNOWN } } }
I20260620 21:50:51.312223 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.308386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.0.193.195:41167)
0620 21:50:51.308505 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.308509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.308531 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:50:51.308660 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.308672 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.308729 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.308822 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:51.309026 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.309565 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.310631 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.310866 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.310956 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.311325 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:51.311797 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:51.311887 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:51.312007 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 21:50:51.312104 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 21:50:51.318668 8460 raft_consensus.cc:1261] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Refusing update from remote peer 6c47a35e5e3942b194046a14d57441df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:51.318677 8600 raft_consensus.cc:1261] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Refusing update from remote peer 6c47a35e5e3942b194046a14d57441df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:51.319465 8821 consensus_queue.cc:1035] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:51.319712 8824 consensus_queue.cc:1035] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [LEADER]: Connected to new peer: Peer: permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:51.330431 8828 mvcc.cc:205] Tried to move back new op lower bound from 7299040261392629760 to 7299040261261885440. Current Snapshot: MvccSnapshot[applied={T|T < 7299040261392629760}]
I20260620 21:50:51.338742 8302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:51.338908 8302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:51.341787 8302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db: Sending DeleteTablet for 3 replicas of tablet 8e3ba5b6f86441229b4b7a4cc03d694a
I20260620 21:50:51.342854 8580 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3ba5b6f86441229b4b7a4cc03d694a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:51 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:43778
I20260620 21:50:51.342907 8440 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3ba5b6f86441229b4b7a4cc03d694a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:51 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:56910
I20260620 21:50:51.343367 8720 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3ba5b6f86441229b4b7a4cc03d694a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:51 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:47326
I20260620 21:50:51.343443 8837 tablet_replica.cc:330] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a: stopping tablet replica
I20260620 21:50:51.343813 8837 raft_consensus.cc:2229] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:51.343729 8302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 21:50:51.344053 8838 tablet_replica.cc:330] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df: stopping tablet replica
I20260620 21:50:51.344188 8837 raft_consensus.cc:2258] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 21:50:51.344185 8302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:51.344379 8838 raft_consensus.cc:2229] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:50:51.344854 8838 raft_consensus.cc:2258] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:51.345212 8839 tablet_replica.cc:330] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee: stopping tablet replica
I20260620 21:50:51.345485 8839 raft_consensus.cc:2229] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:51.345886 8839 raft_consensus.cc:2258] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:51.347415 8838 ts_tablet_manager.cc:1905] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:51.349608 8440 tablet_service.cc:1467] Processing CreateTablet for tablet 48926fff8ffa4cf685b0aa3b74005dcb (DEFAULT_TABLE table=test-table [id=caad947f08d64088af4a90b5ef446615]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:51.349749 8720 tablet_service.cc:1467] Processing CreateTablet for tablet 48926fff8ffa4cf685b0aa3b74005dcb (DEFAULT_TABLE table=test-table [id=caad947f08d64088af4a90b5ef446615]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:51.349915 8440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48926fff8ffa4cf685b0aa3b74005dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:51.349993 8720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48926fff8ffa4cf685b0aa3b74005dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:51.350324 8838 ts_tablet_manager.cc:1918] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:51.350407 8838 log.cc:1196] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/8e3ba5b6f86441229b4b7a4cc03d694a
I20260620 21:50:51.350718 8838 ts_tablet_manager.cc:1939] T 8e3ba5b6f86441229b4b7a4cc03d694a P 6c47a35e5e3942b194046a14d57441df: Deleting consensus metadata
I20260620 21:50:51.352116 8816 tablet_bootstrap.cc:492] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df: Bootstrap starting.
I20260620 21:50:51.352212 8289 catalog_manager.cc:4841] TS 6c47a35e5e3942b194046a14d57441df (127.0.193.195:41167): tablet 8e3ba5b6f86441229b4b7a4cc03d694a (table test-table [id=d81f44ddff52483faf1777f9caf5a280]) successfully deleted
I20260620 21:50:51.353005 8580 tablet_service.cc:1467] Processing CreateTablet for tablet 48926fff8ffa4cf685b0aa3b74005dcb (DEFAULT_TABLE table=test-table [id=caad947f08d64088af4a90b5ef446615]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:51.353116 8816 tablet_bootstrap.cc:654] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:51.353196 8837 ts_tablet_manager.cc:1905] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:51.353256 8580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48926fff8ffa4cf685b0aa3b74005dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:51.353718 8815 tablet_bootstrap.cc:492] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a: Bootstrap starting.
I20260620 21:50:51.353796 8839 ts_tablet_manager.cc:1905] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:51.354712 8815 tablet_bootstrap.cc:654] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:51.356202 8839 ts_tablet_manager.cc:1918] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:51.356282 8814 tablet_bootstrap.cc:492] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee: Bootstrap starting.
I20260620 21:50:51.356297 8839 log.cc:1196] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/8e3ba5b6f86441229b4b7a4cc03d694a
I20260620 21:50:51.356606 8839 ts_tablet_manager.cc:1939] T 8e3ba5b6f86441229b4b7a4cc03d694a P 10b199fe0bfd4600afdd04c76a0583ee: Deleting consensus metadata
I20260620 21:50:51.356614 8837 ts_tablet_manager.cc:1918] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:51.356700 8837 log.cc:1196] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/wal/wals/8e3ba5b6f86441229b4b7a4cc03d694a
I20260620 21:50:51.356983 8837 ts_tablet_manager.cc:1939] T 8e3ba5b6f86441229b4b7a4cc03d694a P 969bc97c9ce149a79d2697b759b4c45a: Deleting consensus metadata
I20260620 21:50:51.357213 8814 tablet_bootstrap.cc:654] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:51.358068 8287 catalog_manager.cc:4841] TS 10b199fe0bfd4600afdd04c76a0583ee (127.0.193.194:43733): tablet 8e3ba5b6f86441229b4b7a4cc03d694a (table test-table [id=d81f44ddff52483faf1777f9caf5a280]) successfully deleted
I20260620 21:50:51.358345 8289 catalog_manager.cc:4841] TS 969bc97c9ce149a79d2697b759b4c45a (127.0.193.193:46347): tablet 8e3ba5b6f86441229b4b7a4cc03d694a (table test-table [id=d81f44ddff52483faf1777f9caf5a280]) successfully deleted
I20260620 21:50:51.359148 8816 tablet_bootstrap.cc:492] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df: No bootstrap required, opened a new log
I20260620 21:50:51.359162 8814 tablet_bootstrap.cc:492] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee: No bootstrap required, opened a new log
I20260620 21:50:51.359288 8814 ts_tablet_manager.cc:1397] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 21:50:51.359288 8816 ts_tablet_manager.cc:1397] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260620 21:50:51.359495 8815 tablet_bootstrap.cc:492] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a: No bootstrap required, opened a new log
I20260620 21:50:51.359586 8815 ts_tablet_manager.cc:1397] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260620 21:50:51.359822 8814 raft_consensus.cc:348] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.359853 8816 raft_consensus.cc:348] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.359977 8816 raft_consensus.cc:374] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:51.359968 8814 raft_consensus.cc:374] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:51.360023 8816 raft_consensus.cc:729] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c47a35e5e3942b194046a14d57441df, State: Initialized, Role: FOLLOWER
I20260620 21:50:51.360021 8814 raft_consensus.cc:729] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b199fe0bfd4600afdd04c76a0583ee, State: Initialized, Role: FOLLOWER
I20260620 21:50:51.360145 8814 consensus_queue.cc:260] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.360157 8816 consensus_queue.cc:260] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.360347 8815 raft_consensus.cc:348] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.360424 8816 ts_tablet_manager.cc:1428] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:50:51.360422 8814 ts_tablet_manager.cc:1428] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:50:51.360452 8815 raft_consensus.cc:374] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:51.360484 8815 raft_consensus.cc:729] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969bc97c9ce149a79d2697b759b4c45a, State: Initialized, Role: FOLLOWER
I20260620 21:50:51.360622 8815 consensus_queue.cc:260] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.360899 8815 ts_tablet_manager.cc:1428] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260620 21:50:51.386929 8787 tablet.cc:2369] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:50:51.392972 8647 tablet.cc:2369] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:50:51.412957 8507 tablet.cc:2369] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:50:51.712931 8820 raft_consensus.cc:482] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:51.713111 8820 raft_consensus.cc:504] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.714354 8820 leader_election.cc:288] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b199fe0bfd4600afdd04c76a0583ee (127.0.193.194:43733), 6c47a35e5e3942b194046a14d57441df (127.0.193.195:41167)
I20260620 21:50:51.719990 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.714638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:43733 (local address 127.0.193.193:56271)
0620 21:50:51.714959 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:51.714980 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:51.715106 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:51.715499 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:51.715503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:51.715525 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:51.715672 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.715679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.716735 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.716738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.718614 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.718627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.719420 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.719424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.719566 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.719586 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 21:50:51.719696 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 21:50:51.719780 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":201,"thread_start_us":83,"threads_started":1}
I20260620 21:50:51.720165 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.714644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:41167 (local address 127.0.193.193:59943)
0620 21:50:51.715469 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:51.715486 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:51.715611 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:51.716016 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:51.716019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:51.716067 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:51.716256 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.716264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.717641 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.717645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.718895 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.718905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.719699 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.719704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.719800 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.719821 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:50:51.719901 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 21:50:51.719959 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":381,"thread_start_us":104,"threads_started":1}
I20260620 21:50:51.720229 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.714824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:59943 (local address 127.0.193.195:41167)
0620 21:50:51.714978 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.714983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.715590 (+ 607us) server_negotiation.cc:407] Connection header received
0620 21:50:51.715726 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.715733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.715833 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.715933 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:51.716391 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.717256 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.719024 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.719554 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.719692 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.719744 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:51.719952 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 21:50:51.720025 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:50:51.720544 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.714866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:56271 (local address 127.0.193.194:43733)
0620 21:50:51.715043 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.715050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.715069 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:50:51.715232 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.715235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.715288 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.715403 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:51.715810 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.716601 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.718775 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.719283 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.719431 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.719499 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:51.720357 (+ 858us) server_negotiation.cc:299] Negotiation successful
0620 21:50:51.720447 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 21:50:51.720950 8600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48926fff8ffa4cf685b0aa3b74005dcb" candidate_uuid: "969bc97c9ce149a79d2697b759b4c45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b199fe0bfd4600afdd04c76a0583ee" is_pre_election: true
I20260620 21:50:51.721047 8740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48926fff8ffa4cf685b0aa3b74005dcb" candidate_uuid: "969bc97c9ce149a79d2697b759b4c45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c47a35e5e3942b194046a14d57441df" is_pre_election: true
I20260620 21:50:51.721146 8600 raft_consensus.cc:2454] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 969bc97c9ce149a79d2697b759b4c45a in term 0.
I20260620 21:50:51.721341 8740 raft_consensus.cc:2454] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 969bc97c9ce149a79d2697b759b4c45a in term 0.
I20260620 21:50:51.721554 8391 leader_election.cc:302] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b199fe0bfd4600afdd04c76a0583ee, 969bc97c9ce149a79d2697b759b4c45a; no voters:
I20260620 21:50:51.721861 8820 raft_consensus.cc:2781] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:51.721928 8820 raft_consensus.cc:482] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:51.721971 8820 raft_consensus.cc:3037] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:51.723244 8820 raft_consensus.cc:504] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.723917 8820 leader_election.cc:288] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [CANDIDATE]: Term 1 election: Requested vote from peers 10b199fe0bfd4600afdd04c76a0583ee (127.0.193.194:43733), 6c47a35e5e3942b194046a14d57441df (127.0.193.195:41167)
I20260620 21:50:51.724325 8600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48926fff8ffa4cf685b0aa3b74005dcb" candidate_uuid: "969bc97c9ce149a79d2697b759b4c45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b199fe0bfd4600afdd04c76a0583ee"
I20260620 21:50:51.724325 8740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48926fff8ffa4cf685b0aa3b74005dcb" candidate_uuid: "969bc97c9ce149a79d2697b759b4c45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c47a35e5e3942b194046a14d57441df"
I20260620 21:50:51.724429 8600 raft_consensus.cc:3037] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:51.724466 8740 raft_consensus.cc:3037] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:51.725421 8600 raft_consensus.cc:2454] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 969bc97c9ce149a79d2697b759b4c45a in term 1.
I20260620 21:50:51.725714 8391 leader_election.cc:302] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b199fe0bfd4600afdd04c76a0583ee, 969bc97c9ce149a79d2697b759b4c45a; no voters:
I20260620 21:50:51.725879 8820 raft_consensus.cc:2781] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:51.725911 8740 raft_consensus.cc:2454] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 969bc97c9ce149a79d2697b759b4c45a in term 1.
I20260620 21:50:51.726049 8820 raft_consensus.cc:686] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [term 1 LEADER]: Becoming Leader. State: Replica: 969bc97c9ce149a79d2697b759b4c45a, State: Running, Role: LEADER
I20260620 21:50:51.726346 8820 consensus_queue.cc:237] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.728941 8302 catalog_manager.cc:5495] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a reported cstate change: term changed from 0 to 1, leader changed from <none> to 969bc97c9ce149a79d2697b759b4c45a (127.0.193.193). New cstate: current_term: 1 leader_uuid: "969bc97c9ce149a79d2697b759b4c45a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } health_report { overall_health: UNKNOWN } } }
Jun 20 21:50:51 dist-test-slave-kk5c krb5kdc[8258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992250, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.254@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
I20260620 21:50:51.839721 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.829417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:33053 (local address 127.0.0.1:37466)
0620 21:50:51.829790 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:51.829804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:51.829912 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:51.830257 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:51.830260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:51.830890 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:51.831075 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.831083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.832092 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.832094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.832702 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.832711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.833203 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.833206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.833278 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.833919 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:51.833931 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:51.835774 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:51.837917 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:51.837932 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:51.837942 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:51.838205 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:51.838510 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:51.838513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:51.838517 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:51.838596 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:51.839017 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:51.839028 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:51.839150 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 21:50:51.839307 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 21:50:51.839582 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":102,"thread_start_us":107,"threads_started":1}
I20260620 21:50:51.839722 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.829505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.0.193.254:33053)
0620 21:50:51.829818 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.829825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.829860 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:50:51.830059 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.830061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.830105 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.830192 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:51.831269 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.831933 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.832846 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.833085 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.833199 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.835956 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:51.835960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:51.835963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:51.835999 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:51.837754 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:51.838317 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:51.838323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:51.838325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:51.838408 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:51.838704 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:51.838707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:51.838709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:51.838907 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:51.839017 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:51.839354 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 21:50:51.839509 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
I20260620 21:50:51.844336 8302 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:37466:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 21:50:51.845064 8302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:51.850435 8580 tablet_service.cc:1467] Processing CreateTablet for tablet c3f3030099ab445ea2f64ce7fa3efeb1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbc5a75acf7d48efa725b69ff46db8d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:50:51.850441 8440 tablet_service.cc:1467] Processing CreateTablet for tablet c3f3030099ab445ea2f64ce7fa3efeb1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbc5a75acf7d48efa725b69ff46db8d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:50:51.850430 8720 tablet_service.cc:1467] Processing CreateTablet for tablet c3f3030099ab445ea2f64ce7fa3efeb1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbc5a75acf7d48efa725b69ff46db8d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 21:50:51.850868 8720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f3030099ab445ea2f64ce7fa3efeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:51.850868 8580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f3030099ab445ea2f64ce7fa3efeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:51.850869 8440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f3030099ab445ea2f64ce7fa3efeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:51.853678 8816 tablet_bootstrap.cc:492] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df: Bootstrap starting.
I20260620 21:50:51.853696 8814 tablet_bootstrap.cc:492] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee: Bootstrap starting.
I20260620 21:50:51.854020 8815 tablet_bootstrap.cc:492] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a: Bootstrap starting.
I20260620 21:50:51.854666 8816 tablet_bootstrap.cc:654] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:51.854689 8814 tablet_bootstrap.cc:654] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:51.854965 8815 tablet_bootstrap.cc:654] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:51.855856 8816 tablet_bootstrap.cc:492] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df: No bootstrap required, opened a new log
I20260620 21:50:51.855948 8816 ts_tablet_manager.cc:1397] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 21:50:51.856236 8815 tablet_bootstrap.cc:492] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a: No bootstrap required, opened a new log
I20260620 21:50:51.856319 8815 ts_tablet_manager.cc:1397] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 21:50:51.856393 8816 raft_consensus.cc:348] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.856529 8816 raft_consensus.cc:374] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:51.856573 8816 raft_consensus.cc:729] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c47a35e5e3942b194046a14d57441df, State: Initialized, Role: FOLLOWER
I20260620 21:50:51.856716 8816 consensus_queue.cc:260] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.856742 8815 raft_consensus.cc:348] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.856875 8815 raft_consensus.cc:374] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:51.856909 8815 raft_consensus.cc:729] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969bc97c9ce149a79d2697b759b4c45a, State: Initialized, Role: FOLLOWER
I20260620 21:50:51.856982 8816 ts_tablet_manager.cc:1428] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 21:50:51.856932 8832 tablet_replica.cc:439] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } }
I20260620 21:50:51.857053 8832 tablet_replica.cc:442] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:51.857023 8815 consensus_queue.cc:260] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.857283 8815 ts_tablet_manager.cc:1428] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:50:51.857487 8814 tablet_bootstrap.cc:492] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee: No bootstrap required, opened a new log
I20260620 21:50:51.857595 8814 ts_tablet_manager.cc:1397] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 21:50:51.858070 8814 raft_consensus.cc:348] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.858204 8814 raft_consensus.cc:374] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:51.858234 8814 raft_consensus.cc:729] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b199fe0bfd4600afdd04c76a0583ee, State: Initialized, Role: FOLLOWER
I20260620 21:50:51.858350 8814 consensus_queue.cc:260] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.858610 8814 ts_tablet_manager.cc:1428] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:51.858790 8820 tablet_replica.cc:439] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } }
I20260620 21:50:51.858929 8820 tablet_replica.cc:442] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:51.860389 8822 tablet_replica.cc:439] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } }
I20260620 21:50:51.860545 8822 tablet_replica.cc:442] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:51.972397 8822 raft_consensus.cc:482] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:51.972575 8822 raft_consensus.cc:504] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.973721 8822 leader_election.cc:288] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 969bc97c9ce149a79d2697b759b4c45a (127.0.193.193:46347), 6c47a35e5e3942b194046a14d57441df (127.0.193.195:41167)
I20260620 21:50:51.977833 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.974560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:40499 (local address 127.0.193.195:41167)
0620 21:50:51.974683 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.974688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.974823 (+ 135us) server_negotiation.cc:407] Connection header received
0620 21:50:51.974980 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.974983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.975040 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.975124 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:51.975383 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.975875 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.976866 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.977273 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.977397 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.977447 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:51.977641 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 21:50:51.977708 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:50:51.977831 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.974223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:52657 (local address 127.0.193.193:46347)
0620 21:50:51.974473 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:51.974477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:51.974495 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:50:51.974604 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:51.974608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:51.974669 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:51.974769 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:50:51.975115 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.975699 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.976793 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.977234 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.977367 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.977427 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:51.977604 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 21:50:51.977681 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":59,"threads_started":1}
I20260620 21:50:51.977864 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.974058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:46347 (local address 127.0.193.194:52657)
0620 21:50:51.974340 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:51.974353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:51.974488 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:51.974796 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:51.974800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:51.974816 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:51.974951 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.974956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.975816 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.975820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.976670 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.976678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.977359 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.977363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.977463 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.977494 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 21:50:51.977586 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 21:50:51.977659 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":23,"spinlock_wait_cycles":19968,"thread_start_us":99,"threads_started":1}
I20260620 21:50:51.977867 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:51.974441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:41167 (local address 127.0.193.194:40499)
0620 21:50:51.974733 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:51.974745 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:51.974873 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:51.975135 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:51.975138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:51.975153 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:50:51.975274 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.975279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.975990 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.975995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.976742 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:51.976750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:51.977398 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:51.977401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:51.977507 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:51.977537 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 21:50:51.977625 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 21:50:51.977695 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"spinlock_wait_cycles":768,"thread_start_us":52,"threads_started":1}
I20260620 21:50:51.978369 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f3030099ab445ea2f64ce7fa3efeb1" candidate_uuid: "10b199fe0bfd4600afdd04c76a0583ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969bc97c9ce149a79d2697b759b4c45a" is_pre_election: true
I20260620 21:50:51.978534 8460 raft_consensus.cc:2454] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b199fe0bfd4600afdd04c76a0583ee in term 0.
I20260620 21:50:51.978569 8740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f3030099ab445ea2f64ce7fa3efeb1" candidate_uuid: "10b199fe0bfd4600afdd04c76a0583ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c47a35e5e3942b194046a14d57441df" is_pre_election: true
I20260620 21:50:51.978729 8740 raft_consensus.cc:2454] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b199fe0bfd4600afdd04c76a0583ee in term 0.
I20260620 21:50:51.978878 8533 leader_election.cc:302] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b199fe0bfd4600afdd04c76a0583ee, 969bc97c9ce149a79d2697b759b4c45a; no voters:
I20260620 21:50:51.979151 8822 raft_consensus.cc:2781] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:51.979238 8822 raft_consensus.cc:482] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:51.979286 8822 raft_consensus.cc:3037] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:51.980319 8822 raft_consensus.cc:504] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.980708 8822 leader_election.cc:288] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [CANDIDATE]: Term 1 election: Requested vote from peers 969bc97c9ce149a79d2697b759b4c45a (127.0.193.193:46347), 6c47a35e5e3942b194046a14d57441df (127.0.193.195:41167)
I20260620 21:50:51.981108 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f3030099ab445ea2f64ce7fa3efeb1" candidate_uuid: "10b199fe0bfd4600afdd04c76a0583ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969bc97c9ce149a79d2697b759b4c45a"
I20260620 21:50:51.981109 8740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f3030099ab445ea2f64ce7fa3efeb1" candidate_uuid: "10b199fe0bfd4600afdd04c76a0583ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c47a35e5e3942b194046a14d57441df"
I20260620 21:50:51.981212 8460 raft_consensus.cc:3037] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:51.981244 8740 raft_consensus.cc:3037] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:51.982116 8460 raft_consensus.cc:2454] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b199fe0bfd4600afdd04c76a0583ee in term 1.
I20260620 21:50:51.982196 8740 raft_consensus.cc:2454] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b199fe0bfd4600afdd04c76a0583ee in term 1.
I20260620 21:50:51.982450 8533 leader_election.cc:302] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b199fe0bfd4600afdd04c76a0583ee, 969bc97c9ce149a79d2697b759b4c45a; no voters:
I20260620 21:50:51.982630 8822 raft_consensus.cc:2781] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:51.982862 8822 raft_consensus.cc:686] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [term 1 LEADER]: Becoming Leader. State: Replica: 10b199fe0bfd4600afdd04c76a0583ee, State: Running, Role: LEADER
I20260620 21:50:51.983120 8822 consensus_queue.cc:237] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:51.984315 8860 tablet_replica.cc:439] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee: TxnStatusTablet state changed. Reason: New leader 10b199fe0bfd4600afdd04c76a0583ee. Latest consensus state: current_term: 1 leader_uuid: "10b199fe0bfd4600afdd04c76a0583ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } }
I20260620 21:50:51.984452 8860 tablet_replica.cc:442] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:50:51.984706 8862 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 21:50:51.984901 8862 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 21:50:51.985634 8302 catalog_manager.cc:5495] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b199fe0bfd4600afdd04c76a0583ee (127.0.193.194). New cstate: current_term: 1 leader_uuid: "10b199fe0bfd4600afdd04c76a0583ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } health_report { overall_health: UNKNOWN } } }
I20260620 21:50:52.013691 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:52.009840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:43733 (local address 127.0.0.1:43786)
0620 21:50:52.010009 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:52.010033 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:52.010157 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:52.010490 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:52.010493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:52.010500 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:50:52.010605 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:52.010611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.011433 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.011437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:52.012046 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:52.012061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.012528 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.012532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:52.012624 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:52.012678 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:50:52.013377 (+ 699us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:50:52.013394 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 21:50:52.013460 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 21:50:52.013560 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 21:50:52.013691 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:52.009932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.0.193.194:43733)
0620 21:50:52.010065 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:52.010071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:52.010088 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:50:52.010298 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:52.010301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:52.010371 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:52.010477 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:52.010715 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.011320 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.012193 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.012415 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.012526 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:52.012825 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:52.013271 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:52.013398 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:52.013481 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 21:50:52.013538 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:50:52.016855 8460 raft_consensus.cc:1261] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Refusing update from remote peer 10b199fe0bfd4600afdd04c76a0583ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:52.017066 8740 raft_consensus.cc:1261] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Refusing update from remote peer 10b199fe0bfd4600afdd04c76a0583ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:52.017441 8860 consensus_queue.cc:1035] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:52.017602 8822 consensus_queue.cc:1035] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:52.019375 8832 tablet_replica.cc:439] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df: TxnStatusTablet state changed. Reason: New leader 10b199fe0bfd4600afdd04c76a0583ee. Latest consensus state: current_term: 1 leader_uuid: "10b199fe0bfd4600afdd04c76a0583ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } }
I20260620 21:50:52.019492 8832 tablet_replica.cc:442] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:52.019094 8820 tablet_replica.cc:439] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a: TxnStatusTablet state changed. Reason: New leader 10b199fe0bfd4600afdd04c76a0583ee. Latest consensus state: current_term: 1 leader_uuid: "10b199fe0bfd4600afdd04c76a0583ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } }
I20260620 21:50:52.019548 8820 tablet_replica.cc:442] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:52.022395 8822 tablet_replica.cc:439] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10b199fe0bfd4600afdd04c76a0583ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } }
I20260620 21:50:52.022603 8822 tablet_replica.cc:442] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:50:52.022773 8832 tablet_replica.cc:439] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10b199fe0bfd4600afdd04c76a0583ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } }
I20260620 21:50:52.022889 8832 tablet_replica.cc:442] T c3f3030099ab445ea2f64ce7fa3efeb1 P 6c47a35e5e3942b194046a14d57441df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:52.023000 8820 tablet_replica.cc:439] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10b199fe0bfd4600afdd04c76a0583ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } }
I20260620 21:50:52.023114 8820 tablet_replica.cc:442] T c3f3030099ab445ea2f64ce7fa3efeb1 P 969bc97c9ce149a79d2697b759b4c45a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 21:50:52.024479 8870 tablet_replica.cc:439] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10b199fe0bfd4600afdd04c76a0583ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } } }
I20260620 21:50:52.024641 8870 tablet_replica.cc:442] T c3f3030099ab445ea2f64ce7fa3efeb1 P 10b199fe0bfd4600afdd04c76a0583ee: This TxnStatusTablet replica's current role is: LEADER
I20260620 21:50:52.024937 8866 mvcc.cc:205] Tried to move back new op lower bound from 7299040264255115264 to 7299040264125669376. Current Snapshot: MvccSnapshot[applied={T|T < 7299040264255115264}]
I20260620 21:50:52.034722 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:52.030791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.0.193.193:46347)
0620 21:50:52.030918 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:52.030923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:52.031023 (+ 100us) server_negotiation.cc:407] Connection header received
0620 21:50:52.031253 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:52.031256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:52.031323 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:52.031439 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:52.031685 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.032245 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.033172 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.033442 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.033500 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:52.033838 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:52.034316 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:52.034447 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:52.034509 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 21:50:52.034571 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:50:52.046792 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:52.043002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:56117 (local address 127.0.193.194:43733)
0620 21:50:52.043158 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:52.043165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:52.043355 (+ 190us) server_negotiation.cc:407] Connection header received
0620 21:50:52.043554 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:52.043556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:52.043603 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:52.043664 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:52.043981 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.044487 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.045783 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.045991 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.046067 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:52.046387 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:52.046455 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:52.046510 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:52.046639 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 21:50:52.046681 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:50:52.046862 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:52.042908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:43733 (local address 127.0.193.193:56117)
0620 21:50:52.043266 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:52.043282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:52.043433 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:52.043725 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:52.043728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:52.043737 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:50:52.043868 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:52.043874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.044599 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.044602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:52.045659 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:52.045669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.046104 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.046108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:52.046203 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:52.046259 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:50:52.046549 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:50:52.046555 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:50:52.046621 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 21:50:52.046699 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":123,"threads_started":1}
I20260620 21:50:52.054136 8600 raft_consensus.cc:1261] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Refusing update from remote peer 969bc97c9ce149a79d2697b759b4c45a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:52.054247 8740 raft_consensus.cc:1261] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Refusing update from remote peer 969bc97c9ce149a79d2697b759b4c45a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:52.054710 8820 consensus_queue.cc:1035] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:52.054972 8845 consensus_queue.cc:1035] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:52.058782 8867 mvcc.cc:205] Tried to move back new op lower bound from 7299040264409288704 to 7299040263074209792. Current Snapshot: MvccSnapshot[applied={T|T < 7299040264409288704}]
I20260620 21:50:52.079432 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:52.075303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:50583 (local address 127.0.193.193:46347)
0620 21:50:52.075419 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:52.075424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:52.075759 (+ 335us) server_negotiation.cc:407] Connection header received
0620 21:50:52.075886 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:52.075888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:52.075933 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:52.075993 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:52.076272 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.076757 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.078194 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.078480 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.078611 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:52.078930 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:52.079005 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:52.079095 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:52.079223 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 21:50:52.079278 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:50:52.079447 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:52.075159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:46347 (local address 127.0.193.194:50583)
0620 21:50:52.075677 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:52.075694 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:52.075801 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:52.076037 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:52.076040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:52.076047 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 21:50:52.076173 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:52.076179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.076856 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.076860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:52.078038 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:52.078049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.078601 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.078605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:52.078705 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:52.078761 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 21:50:52.079102 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 21:50:52.079111 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 21:50:52.079207 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 21:50:52.079290 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":111,"threads_started":1}
I20260620 21:50:52.109733 8303 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:52.109934 8303 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:52.112684 8303 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db: Sending DeleteTablet for 3 replicas of tablet 48926fff8ffa4cf685b0aa3b74005dcb
I20260620 21:50:52.113430 8720 tablet_service.cc:1514] Processing DeleteTablet for tablet 48926fff8ffa4cf685b0aa3b74005dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:52 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:47326
I20260620 21:50:52.113545 8580 tablet_service.cc:1514] Processing DeleteTablet for tablet 48926fff8ffa4cf685b0aa3b74005dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:52 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:43778
I20260620 21:50:52.113909 8887 tablet_replica.cc:330] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee: stopping tablet replica
I20260620 21:50:52.113960 8440 tablet_service.cc:1514] Processing DeleteTablet for tablet 48926fff8ffa4cf685b0aa3b74005dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:52 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:56910
I20260620 21:50:52.114042 8887 raft_consensus.cc:2229] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:52.114159 8887 raft_consensus.cc:2258] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:52.114883 8887 ts_tablet_manager.cc:1905] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:52.118083 8887 ts_tablet_manager.cc:1918] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:50:52.118172 8887 log.cc:1196] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/48926fff8ffa4cf685b0aa3b74005dcb
I20260620 21:50:52.118489 8887 ts_tablet_manager.cc:1939] T 48926fff8ffa4cf685b0aa3b74005dcb P 10b199fe0bfd4600afdd04c76a0583ee: Deleting consensus metadata
I20260620 21:50:52.119326 8287 catalog_manager.cc:4841] TS 10b199fe0bfd4600afdd04c76a0583ee (127.0.193.194:43733): tablet 48926fff8ffa4cf685b0aa3b74005dcb (table test-table [id=caad947f08d64088af4a90b5ef446615]) successfully deleted
I20260620 21:50:52.121750 8886 tablet_replica.cc:330] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df: stopping tablet replica
I20260620 21:50:52.122026 8888 tablet_replica.cc:330] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a: stopping tablet replica
I20260620 21:50:52.122228 8888 raft_consensus.cc:2229] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:50:52.122402 8886 raft_consensus.cc:2229] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:52.122654 8886 raft_consensus.cc:2258] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:52.123122 8888 raft_consensus.cc:2258] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:52.124614 8888 ts_tablet_manager.cc:1905] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:52.124782 8886 ts_tablet_manager.cc:1905] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:52.130470 8888 ts_tablet_manager.cc:1918] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:50:52.130661 8888 log.cc:1196] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/wal/wals/48926fff8ffa4cf685b0aa3b74005dcb
I20260620 21:50:52.131007 8888 ts_tablet_manager.cc:1939] T 48926fff8ffa4cf685b0aa3b74005dcb P 969bc97c9ce149a79d2697b759b4c45a: Deleting consensus metadata
I20260620 21:50:52.132172 8886 ts_tablet_manager.cc:1918] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:50:52.132192 8289 catalog_manager.cc:4841] TS 969bc97c9ce149a79d2697b759b4c45a (127.0.193.193:46347): tablet 48926fff8ffa4cf685b0aa3b74005dcb (table test-table [id=caad947f08d64088af4a90b5ef446615]) successfully deleted
I20260620 21:50:52.132272 8886 log.cc:1196] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/48926fff8ffa4cf685b0aa3b74005dcb
I20260620 21:50:52.132536 8886 ts_tablet_manager.cc:1939] T 48926fff8ffa4cf685b0aa3b74005dcb P 6c47a35e5e3942b194046a14d57441df: Deleting consensus metadata
I20260620 21:50:52.133489 8289 catalog_manager.cc:4841] TS 6c47a35e5e3942b194046a14d57441df (127.0.193.195:41167): tablet 48926fff8ffa4cf685b0aa3b74005dcb (table test-table [id=caad947f08d64088af4a90b5ef446615]) successfully deleted
Jun 20 21:50:52 dist-test-slave-kk5c krb5kdc[8258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 21:50:52 dist-test-slave-kk5c krb5kdc[8258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:52 dist-test-slave-kk5c krb5kdc[8258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 21:50:52.154836 8853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:52.146285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.0.193.254:33053)
0620 21:50:52.146413 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:52.146420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:52.146978 (+ 558us) server_negotiation.cc:407] Connection header received
0620 21:50:52.147181 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:52.147207 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:52.147264 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:52.147319 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:52.148339 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.149179 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.150410 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.150771 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.150842 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:52.154694 (+ 3852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37474: BlockingRecv error: recv got EOF from 127.0.0.1:37474 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 21:50:52 dist-test-slave-kk5c krb5kdc[8258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM
I20260620 21:50:52.169379 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:52.157684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.0.193.254:33053)
0620 21:50:52.157793 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:52.157797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:52.158380 (+ 583us) server_negotiation.cc:407] Connection header received
0620 21:50:52.158566 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:52.158569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:52.158630 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:52.158684 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:52.159631 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.160454 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.161731 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.162065 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.162136 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:52.165419 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:52.165424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:52.165426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:52.165461 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:52.167286 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:52.168033 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:52.168036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:52.168038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:52.168133 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:52.168416 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:52.168419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:52.168421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:52.168657 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:52.168788 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:52.169053 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 21:50:52.169207 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 21:50:52.172652 8303 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:50:52.173188 8303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:52.178275 8580 tablet_service.cc:1467] Processing CreateTablet for tablet a54add22ecf44d8191eda86e9ba18745 (DEFAULT_TABLE table=test-table [id=da9eb5d00323401595b79e527db1d4cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:52.178275 8440 tablet_service.cc:1467] Processing CreateTablet for tablet a54add22ecf44d8191eda86e9ba18745 (DEFAULT_TABLE table=test-table [id=da9eb5d00323401595b79e527db1d4cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:52.178495 8720 tablet_service.cc:1467] Processing CreateTablet for tablet a54add22ecf44d8191eda86e9ba18745 (DEFAULT_TABLE table=test-table [id=da9eb5d00323401595b79e527db1d4cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:52.178610 8440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54add22ecf44d8191eda86e9ba18745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:52.178610 8580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54add22ecf44d8191eda86e9ba18745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:52.178800 8720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54add22ecf44d8191eda86e9ba18745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:52.181223 8814 tablet_bootstrap.cc:492] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee: Bootstrap starting.
I20260620 21:50:52.181308 8815 tablet_bootstrap.cc:492] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a: Bootstrap starting.
I20260620 21:50:52.181983 8814 tablet_bootstrap.cc:654] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:52.182274 8815 tablet_bootstrap.cc:654] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:52.182870 8816 tablet_bootstrap.cc:492] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df: Bootstrap starting.
I20260620 21:50:52.183131 8814 tablet_bootstrap.cc:492] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee: No bootstrap required, opened a new log
I20260620 21:50:52.183240 8814 ts_tablet_manager.cc:1397] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:50:52.183578 8814 raft_consensus.cc:348] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.183673 8814 raft_consensus.cc:374] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:52.183696 8814 raft_consensus.cc:729] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b199fe0bfd4600afdd04c76a0583ee, State: Initialized, Role: FOLLOWER
I20260620 21:50:52.183720 8816 tablet_bootstrap.cc:654] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:52.183828 8814 consensus_queue.cc:260] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.184113 8814 ts_tablet_manager.cc:1428] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:50:52.186115 8816 tablet_bootstrap.cc:492] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df: No bootstrap required, opened a new log
I20260620 21:50:52.186213 8816 ts_tablet_manager.cc:1397] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 21:50:52.186610 8816 raft_consensus.cc:348] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.186741 8816 raft_consensus.cc:374] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:52.186774 8816 raft_consensus.cc:729] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c47a35e5e3942b194046a14d57441df, State: Initialized, Role: FOLLOWER
I20260620 21:50:52.186895 8816 consensus_queue.cc:260] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.187155 8816 ts_tablet_manager.cc:1428] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:50:52.187757 8815 tablet_bootstrap.cc:492] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a: No bootstrap required, opened a new log
I20260620 21:50:52.187856 8815 ts_tablet_manager.cc:1397] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260620 21:50:52.188315 8815 raft_consensus.cc:348] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.188428 8815 raft_consensus.cc:374] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:52.188458 8815 raft_consensus.cc:729] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969bc97c9ce149a79d2697b759b4c45a, State: Initialized, Role: FOLLOWER
I20260620 21:50:52.188567 8815 consensus_queue.cc:260] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.188810 8815 ts_tablet_manager.cc:1428] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:52.311687 8832 raft_consensus.cc:482] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:52.311852 8832 raft_consensus.cc:504] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.312292 8832 leader_election.cc:288] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 969bc97c9ce149a79d2697b759b4c45a (127.0.193.193:46347), 10b199fe0bfd4600afdd04c76a0583ee (127.0.193.194:43733)
I20260620 21:50:52.312806 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54add22ecf44d8191eda86e9ba18745" candidate_uuid: "6c47a35e5e3942b194046a14d57441df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969bc97c9ce149a79d2697b759b4c45a" is_pre_election: true
I20260620 21:50:52.312950 8460 raft_consensus.cc:2454] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c47a35e5e3942b194046a14d57441df in term 0.
I20260620 21:50:52.312968 8600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54add22ecf44d8191eda86e9ba18745" candidate_uuid: "6c47a35e5e3942b194046a14d57441df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b199fe0bfd4600afdd04c76a0583ee" is_pre_election: true
I20260620 21:50:52.313146 8600 raft_consensus.cc:2454] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c47a35e5e3942b194046a14d57441df in term 0.
I20260620 21:50:52.313299 8673 leader_election.cc:302] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c47a35e5e3942b194046a14d57441df, 969bc97c9ce149a79d2697b759b4c45a; no voters:
I20260620 21:50:52.313534 8832 raft_consensus.cc:2781] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:52.313582 8832 raft_consensus.cc:482] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:52.313613 8832 raft_consensus.cc:3037] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:52.314885 8832 raft_consensus.cc:504] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.315348 8832 leader_election.cc:288] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [CANDIDATE]: Term 1 election: Requested vote from peers 969bc97c9ce149a79d2697b759b4c45a (127.0.193.193:46347), 10b199fe0bfd4600afdd04c76a0583ee (127.0.193.194:43733)
I20260620 21:50:52.315853 8600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54add22ecf44d8191eda86e9ba18745" candidate_uuid: "6c47a35e5e3942b194046a14d57441df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b199fe0bfd4600afdd04c76a0583ee"
I20260620 21:50:52.315855 8460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54add22ecf44d8191eda86e9ba18745" candidate_uuid: "6c47a35e5e3942b194046a14d57441df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969bc97c9ce149a79d2697b759b4c45a"
I20260620 21:50:52.315994 8460 raft_consensus.cc:3037] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:52.316006 8600 raft_consensus.cc:3037] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:52.317157 8600 raft_consensus.cc:2454] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c47a35e5e3942b194046a14d57441df in term 1.
I20260620 21:50:52.317158 8460 raft_consensus.cc:2454] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c47a35e5e3942b194046a14d57441df in term 1.
I20260620 21:50:52.317476 8673 leader_election.cc:302] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c47a35e5e3942b194046a14d57441df, 969bc97c9ce149a79d2697b759b4c45a; no voters:
I20260620 21:50:52.317694 8832 raft_consensus.cc:2781] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:52.317921 8832 raft_consensus.cc:686] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [term 1 LEADER]: Becoming Leader. State: Replica: 6c47a35e5e3942b194046a14d57441df, State: Running, Role: LEADER
I20260620 21:50:52.318082 8832 consensus_queue.cc:237] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.320112 8301 catalog_manager.cc:5495] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c47a35e5e3942b194046a14d57441df (127.0.193.195). New cstate: current_term: 1 leader_uuid: "6c47a35e5e3942b194046a14d57441df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } health_report { overall_health: HEALTHY } } }
I20260620 21:50:52.347880 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:52.344717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.0.193.195:41167)
0620 21:50:52.344845 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:52.344848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:52.344862 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:50:52.344970 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:52.344972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:52.345015 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:52.345085 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:52.345346 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.345867 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.346824 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.347049 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.347135 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:52.347433 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:52.347502 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:52.347563 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:52.347722 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 21:50:52.347773 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:50:52.350144 8460 raft_consensus.cc:1261] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Refusing update from remote peer 6c47a35e5e3942b194046a14d57441df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:52.350163 8600 raft_consensus.cc:1261] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Refusing update from remote peer 6c47a35e5e3942b194046a14d57441df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:52.350698 8904 consensus_queue.cc:1035] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:52.350878 8832 consensus_queue.cc:1035] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [LEADER]: Connected to new peer: Peer: permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:52.359280 8301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:52.359648 8301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:52.361694 8301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db: Sending DeleteTablet for 3 replicas of tablet a54add22ecf44d8191eda86e9ba18745
I20260620 21:50:52.362344 8440 tablet_service.cc:1514] Processing DeleteTablet for tablet a54add22ecf44d8191eda86e9ba18745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:52 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:56910
I20260620 21:50:52.362473 8720 tablet_service.cc:1514] Processing DeleteTablet for tablet a54add22ecf44d8191eda86e9ba18745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:52 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:47326
I20260620 21:50:52.362576 8580 tablet_service.cc:1514] Processing DeleteTablet for tablet a54add22ecf44d8191eda86e9ba18745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:52 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:43778
I20260620 21:50:52.362797 8887 tablet_replica.cc:330] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee: stopping tablet replica
I20260620 21:50:52.362818 8888 tablet_replica.cc:330] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a: stopping tablet replica
I20260620 21:50:52.362939 8887 raft_consensus.cc:2229] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:52.362948 8888 raft_consensus.cc:2229] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:52.363092 8887 raft_consensus.cc:2258] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:52.363087 8888 raft_consensus.cc:2258] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:52.363282 8886 tablet_replica.cc:330] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df: stopping tablet replica
I20260620 21:50:52.363407 8886 raft_consensus.cc:2229] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:50:52.363396 8301 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 21:50:52.363611 8886 raft_consensus.cc:2258] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 21:50:52.363837 8301 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:50:52.363884 8887 ts_tablet_manager.cc:1905] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:52.364400 8886 ts_tablet_manager.cc:1905] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:52.364430 8888 ts_tablet_manager.cc:1905] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:52.366860 8886 ts_tablet_manager.cc:1918] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:52.366930 8886 log.cc:1196] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/a54add22ecf44d8191eda86e9ba18745
I20260620 21:50:52.367219 8886 ts_tablet_manager.cc:1939] T a54add22ecf44d8191eda86e9ba18745 P 6c47a35e5e3942b194046a14d57441df: Deleting consensus metadata
I20260620 21:50:52.367427 8888 ts_tablet_manager.cc:1918] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:52.367517 8888 log.cc:1196] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-0/wal/wals/a54add22ecf44d8191eda86e9ba18745
I20260620 21:50:52.367754 8888 ts_tablet_manager.cc:1939] T a54add22ecf44d8191eda86e9ba18745 P 969bc97c9ce149a79d2697b759b4c45a: Deleting consensus metadata
I20260620 21:50:52.368629 8289 catalog_manager.cc:4841] TS 969bc97c9ce149a79d2697b759b4c45a (127.0.193.193:46347): tablet a54add22ecf44d8191eda86e9ba18745 (table test-table [id=da9eb5d00323401595b79e527db1d4cb]) successfully deleted
I20260620 21:50:52.371341 8440 tablet_service.cc:1467] Processing CreateTablet for tablet cabaf8821bfd43eea597fa8029af7ab0 (DEFAULT_TABLE table=test-table [id=7a850265a4fa4b18957e65333528f1b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:52.371555 8580 tablet_service.cc:1467] Processing CreateTablet for tablet cabaf8821bfd43eea597fa8029af7ab0 (DEFAULT_TABLE table=test-table [id=7a850265a4fa4b18957e65333528f1b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:52.371646 8440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cabaf8821bfd43eea597fa8029af7ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:52.371852 8580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cabaf8821bfd43eea597fa8029af7ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:52.372376 8887 ts_tablet_manager.cc:1918] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:50:52.372454 8887 log.cc:1196] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/a54add22ecf44d8191eda86e9ba18745
I20260620 21:50:52.372922 8887 ts_tablet_manager.cc:1939] T a54add22ecf44d8191eda86e9ba18745 P 10b199fe0bfd4600afdd04c76a0583ee: Deleting consensus metadata
I20260620 21:50:52.374048 8815 tablet_bootstrap.cc:492] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a: Bootstrap starting.
I20260620 21:50:52.374405 8289 catalog_manager.cc:4841] TS 6c47a35e5e3942b194046a14d57441df (127.0.193.195:41167): tablet a54add22ecf44d8191eda86e9ba18745 (table test-table [id=da9eb5d00323401595b79e527db1d4cb]) successfully deleted
I20260620 21:50:52.374735 8287 catalog_manager.cc:4841] TS 10b199fe0bfd4600afdd04c76a0583ee (127.0.193.194:43733): tablet a54add22ecf44d8191eda86e9ba18745 (table test-table [id=da9eb5d00323401595b79e527db1d4cb]) successfully deleted
I20260620 21:50:52.374993 8814 tablet_bootstrap.cc:492] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee: Bootstrap starting.
I20260620 21:50:52.375000 8720 tablet_service.cc:1467] Processing CreateTablet for tablet cabaf8821bfd43eea597fa8029af7ab0 (DEFAULT_TABLE table=test-table [id=7a850265a4fa4b18957e65333528f1b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:50:52.375066 8815 tablet_bootstrap.cc:654] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:52.375291 8720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cabaf8821bfd43eea597fa8029af7ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:52.375859 8814 tablet_bootstrap.cc:654] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:52.376937 8814 tablet_bootstrap.cc:492] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee: No bootstrap required, opened a new log
I20260620 21:50:52.377043 8814 ts_tablet_manager.cc:1397] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:50:52.377454 8816 tablet_bootstrap.cc:492] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df: Bootstrap starting.
I20260620 21:50:52.377452 8814 raft_consensus.cc:348] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.377563 8814 raft_consensus.cc:374] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:52.377594 8814 raft_consensus.cc:729] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b199fe0bfd4600afdd04c76a0583ee, State: Initialized, Role: FOLLOWER
I20260620 21:50:52.377713 8814 consensus_queue.cc:260] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.377957 8814 ts_tablet_manager.cc:1428] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:52.378260 8816 tablet_bootstrap.cc:654] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df: Neither blocks nor log segments found. Creating new log.
I20260620 21:50:52.379338 8816 tablet_bootstrap.cc:492] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df: No bootstrap required, opened a new log
I20260620 21:50:52.379427 8816 ts_tablet_manager.cc:1397] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 21:50:52.379848 8816 raft_consensus.cc:348] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.379927 8816 raft_consensus.cc:374] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:52.379948 8816 raft_consensus.cc:729] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c47a35e5e3942b194046a14d57441df, State: Initialized, Role: FOLLOWER
I20260620 21:50:52.380070 8816 consensus_queue.cc:260] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.380328 8816 ts_tablet_manager.cc:1428] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 21:50:52.380817 8815 tablet_bootstrap.cc:492] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a: No bootstrap required, opened a new log
I20260620 21:50:52.380913 8815 ts_tablet_manager.cc:1397] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 21:50:52.381248 8815 raft_consensus.cc:348] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.381328 8815 raft_consensus.cc:374] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:52.381349 8815 raft_consensus.cc:729] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969bc97c9ce149a79d2697b759b4c45a, State: Initialized, Role: FOLLOWER
I20260620 21:50:52.381485 8815 consensus_queue.cc:260] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.381785 8815 ts_tablet_manager.cc:1428] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:50:52.409307 8881 raft_consensus.cc:482] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:50:52.409489 8881 raft_consensus.cc:504] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.410378 8881 leader_election.cc:288] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b199fe0bfd4600afdd04c76a0583ee (127.0.193.194:43733), 6c47a35e5e3942b194046a14d57441df (127.0.193.195:41167)
I20260620 21:50:52.410878 8600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabaf8821bfd43eea597fa8029af7ab0" candidate_uuid: "969bc97c9ce149a79d2697b759b4c45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b199fe0bfd4600afdd04c76a0583ee" is_pre_election: true
I20260620 21:50:52.411051 8740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabaf8821bfd43eea597fa8029af7ab0" candidate_uuid: "969bc97c9ce149a79d2697b759b4c45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c47a35e5e3942b194046a14d57441df" is_pre_election: true
I20260620 21:50:52.411113 8600 raft_consensus.cc:2454] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 969bc97c9ce149a79d2697b759b4c45a in term 0.
I20260620 21:50:52.411259 8740 raft_consensus.cc:2454] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 969bc97c9ce149a79d2697b759b4c45a in term 0.
I20260620 21:50:52.411574 8391 leader_election.cc:302] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b199fe0bfd4600afdd04c76a0583ee, 969bc97c9ce149a79d2697b759b4c45a; no voters:
I20260620 21:50:52.411765 8881 raft_consensus.cc:2781] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:50:52.411829 8881 raft_consensus.cc:482] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:50:52.411859 8881 raft_consensus.cc:3037] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:52.413172 8881 raft_consensus.cc:504] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.413673 8881 leader_election.cc:288] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [CANDIDATE]: Term 1 election: Requested vote from peers 10b199fe0bfd4600afdd04c76a0583ee (127.0.193.194:43733), 6c47a35e5e3942b194046a14d57441df (127.0.193.195:41167)
I20260620 21:50:52.414260 8600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabaf8821bfd43eea597fa8029af7ab0" candidate_uuid: "969bc97c9ce149a79d2697b759b4c45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b199fe0bfd4600afdd04c76a0583ee"
I20260620 21:50:52.414258 8740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabaf8821bfd43eea597fa8029af7ab0" candidate_uuid: "969bc97c9ce149a79d2697b759b4c45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c47a35e5e3942b194046a14d57441df"
I20260620 21:50:52.414423 8740 raft_consensus.cc:3037] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:52.414423 8600 raft_consensus.cc:3037] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:52.416050 8600 raft_consensus.cc:2454] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 969bc97c9ce149a79d2697b759b4c45a in term 1.
I20260620 21:50:52.416050 8740 raft_consensus.cc:2454] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 969bc97c9ce149a79d2697b759b4c45a in term 1.
I20260620 21:50:52.416409 8391 leader_election.cc:302] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b199fe0bfd4600afdd04c76a0583ee, 969bc97c9ce149a79d2697b759b4c45a; no voters:
I20260620 21:50:52.416697 8881 raft_consensus.cc:2781] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:52.416828 8881 raft_consensus.cc:686] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [term 1 LEADER]: Becoming Leader. State: Replica: 969bc97c9ce149a79d2697b759b4c45a, State: Running, Role: LEADER
I20260620 21:50:52.417058 8881 consensus_queue.cc:237] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } }
I20260620 21:50:52.419384 8303 catalog_manager.cc:5495] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a reported cstate change: term changed from 0 to 1, leader changed from <none> to 969bc97c9ce149a79d2697b759b4c45a (127.0.193.193). New cstate: current_term: 1 leader_uuid: "969bc97c9ce149a79d2697b759b4c45a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "969bc97c9ce149a79d2697b759b4c45a" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 } health_report { overall_health: UNKNOWN } } }
I20260620 21:50:52.444067 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:52.440858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.0.193.193:46347)
0620 21:50:52.441041 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:52.441045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:52.441058 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:50:52.441105 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:52.441106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:52.441221 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:52.441290 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:50:52.441538 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.442118 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.443026 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:52.443310 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:52.443394 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:52.443624 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:50:52.443703 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:50:52.443758 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:52.443840 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 21:50:52.443888 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 21:50:52.451452 8740 raft_consensus.cc:1261] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Refusing update from remote peer 969bc97c9ce149a79d2697b759b4c45a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:52.451537 8600 raft_consensus.cc:1261] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Refusing update from remote peer 969bc97c9ce149a79d2697b759b4c45a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:50:52.451958 8881 consensus_queue.cc:1035] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c47a35e5e3942b194046a14d57441df" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 41167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:52.452142 8820 consensus_queue.cc:1035] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b199fe0bfd4600afdd04c76a0583ee" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 43733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:50:52.492291 8299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:52.492729 8299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:50:52.495760 8299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92b57d2ea2724612bfe812312cf465db: Sending DeleteTablet for 3 replicas of tablet cabaf8821bfd43eea597fa8029af7ab0
I20260620 21:50:52.496523 8720 tablet_service.cc:1514] Processing DeleteTablet for tablet cabaf8821bfd43eea597fa8029af7ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:52 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:47326
I20260620 21:50:52.496523 8440 tablet_service.cc:1514] Processing DeleteTablet for tablet cabaf8821bfd43eea597fa8029af7ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:52 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:56910
I20260620 21:50:52.496562 8580 tablet_service.cc:1514] Processing DeleteTablet for tablet cabaf8821bfd43eea597fa8029af7ab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:50:52 UTC) from {username='oryx', principal='oryx/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:43778
I20260620 21:50:52.496753 8887 tablet_replica.cc:330] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee: stopping tablet replica
I20260620 21:50:52.496753 8886 tablet_replica.cc:330] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df: stopping tablet replica
I20260620 21:50:52.496774 8888 tablet_replica.cc:330] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a: stopping tablet replica
I20260620 21:50:52.496878 8887 raft_consensus.cc:2229] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:52.496877 8886 raft_consensus.cc:2229] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:50:52.496905 8888 raft_consensus.cc:2229] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:50:52.497010 8887 raft_consensus.cc:2258] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:52.497010 8886 raft_consensus.cc:2258] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:52.497102 8888 raft_consensus.cc:2258] T cabaf8821bfd43eea597fa8029af7ab0 P 969bc97c9ce149a79d2697b759b4c45a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:50:52.497797 8887 ts_tablet_manager.cc:1905] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:52.498009 8886 ts_tablet_manager.cc:1905] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:50:52.498024 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 8375
I20260620 21:50:52.501181 8887 ts_tablet_manager.cc:1918] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:50:52.501178 8886 ts_tablet_manager.cc:1918] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 21:50:52.501303 8887 log.cc:1196] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/cabaf8821bfd43eea597fa8029af7ab0
I20260620 21:50:52.501300 8886 log.cc:1196] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/cabaf8821bfd43eea597fa8029af7ab0
I20260620 21:50:52.501601 8886 ts_tablet_manager.cc:1939] T cabaf8821bfd43eea597fa8029af7ab0 P 6c47a35e5e3942b194046a14d57441df: Deleting consensus metadata
I20260620 21:50:52.501607 8887 ts_tablet_manager.cc:1939] T cabaf8821bfd43eea597fa8029af7ab0 P 10b199fe0bfd4600afdd04c76a0583ee: Deleting consensus metadata
I20260620 21:50:52.502401 8289 catalog_manager.cc:4841] TS 6c47a35e5e3942b194046a14d57441df (127.0.193.195:41167): tablet cabaf8821bfd43eea597fa8029af7ab0 (table test-table [id=7a850265a4fa4b18957e65333528f1b8]) successfully deleted
I20260620 21:50:52.502442 8287 catalog_manager.cc:4841] TS 10b199fe0bfd4600afdd04c76a0583ee (127.0.193.194:43733): tablet cabaf8821bfd43eea597fa8029af7ab0 (table test-table [id=7a850265a4fa4b18957e65333528f1b8]) successfully deleted
W20260620 21:50:52.508345 8289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:46347 (error 108)
I20260620 21:50:52.508519 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 8515
I20260620 21:50:52.510648 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:52.510040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:46347 (local address 127.0.0.1:56926)
0620 21:50:52.510398 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:52.510484 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:46347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":59,"threads_started":1}
W20260620 21:50:52.510816 8289 catalog_manager.cc:4584] TS 969bc97c9ce149a79d2697b759b4c45a (127.0.193.193:46347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cabaf8821bfd43eea597fa8029af7ab0: Network error: Client connection negotiation failed: client connection to 127.0.193.193:46347: connect: Connection refused (error 111)
I20260620 21:50:52.515976 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 8655
I20260620 21:50:52.523306 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 8271
2026-06-20T21:50:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3558 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:50:52 dist-test-slave-kk5c krb5kdc[8919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:50:52 dist-test-slave-kk5c krb5kdc[8919](info): set up 2 sockets
Jun 20 21:50:52 dist-test-slave-kk5c krb5kdc[8919](info): commencing operation
krb5kdc: starting...
W20260620 21:50:53.592111 775 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:50:53 dist-test-slave-kk5c krb5kdc[8919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:50:53.739738 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:37231
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:37231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:53.853790 8932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:53.854413 8932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:53.854564 8932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:53.859105 8932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:53.859215 8932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:53.859238 8932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:53.859256 8932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:53.859272 8932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:53.864843 8932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:37231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:37231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:53.866102 8932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:53.867136 8932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:53.876093 8938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:53.876096 8940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:53.876130 8937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:53.876539 8932 server_base.cc:1034] running on GCE node
I20260620 21:50:53.877123 8932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:53.878342 8932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:53.879580 8932 hybrid_clock.cc:648] HybridClock initialized: now 1781992253879557 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T21:50:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1363 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:50:53 dist-test-slave-kk5c krb5kdc[8945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:50:53 dist-test-slave-kk5c krb5kdc[8945](info): set up 2 sockets
Jun 20 21:50:53 dist-test-slave-kk5c krb5kdc[8945](info): commencing operation
krb5kdc: starting...
W20260620 21:50:54.956091 775 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:50:55 dist-test-slave-kk5c krb5kdc[8945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:50:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:50:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:50:55.090559 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:41369
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:41369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:55.209141 8958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:55.209481 8958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:55.209569 8958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:55.214056 8958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:55.214174 8958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:55.214210 8958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:55.214250 8958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:55.214287 8958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:55.220482 8958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:41369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:41369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:55.221719 8958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:55.222818 8958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:55.232252 8964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:55.232477 8966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:55.232239 8963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:55.232649 8958 server_base.cc:1034] running on GCE node
I20260620 21:50:55.233587 8958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:55.235342 8958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:55.236596 8958 hybrid_clock.cc:648] HybridClock initialized: now 1781992255236636 us; error 117 us; skew 500 ppm
Jun 20 21:50:55 dist-test-slave-kk5c krb5kdc[8945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:55.240132 8958 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:50:55.241418 8958 webserver.cc:460] Webserver started at http://127.0.193.254:37285/ using document root <none> and password file <none>
I20260620 21:50:55.242008 8958 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:55.242075 8958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:55.242254 8958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:55.244927 8958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "b96204bca20746faa1f5421828ec0ea6"
format_stamp: "Formatted at 2026-06-20 21:50:55 on dist-test-slave-kk5c"
server_key: "b5af88cf330f2de14d11295aa67f8df7"
server_key_iv: "d514569b20bfb0ae5414884d55b89385"
server_key_version: "encryptionkey@0"
I20260620 21:50:55.245551 8958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "b96204bca20746faa1f5421828ec0ea6"
format_stamp: "Formatted at 2026-06-20 21:50:55 on dist-test-slave-kk5c"
server_key: "b5af88cf330f2de14d11295aa67f8df7"
server_key_iv: "d514569b20bfb0ae5414884d55b89385"
server_key_version: "encryptionkey@0"
I20260620 21:50:55.249845 8958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:50:55.252492 8973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:55.253839 8958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 21:50:55.253943 8958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "b96204bca20746faa1f5421828ec0ea6"
format_stamp: "Formatted at 2026-06-20 21:50:55 on dist-test-slave-kk5c"
server_key: "b5af88cf330f2de14d11295aa67f8df7"
server_key_iv: "d514569b20bfb0ae5414884d55b89385"
server_key_version: "encryptionkey@0"
I20260620 21:50:55.254038 8958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:55.272725 8958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:55.276072 8958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:55.276240 8958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:55.283812 8958 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:41369
I20260620 21:50:55.283838 9025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:41369 every 8 connection(s)
I20260620 21:50:55.284878 8958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:55.287813 9026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:50:55.288100 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 8958
I20260620 21:50:55.288237 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:50:55.288607 775 external_mini_cluster.cc:1428] Setting key 9f85a2e5192507cb673b03708c55a7dd
I20260620 21:50:55.295351 9026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6: Bootstrap starting.
I20260620 21:50:55.298069 9026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6: Neither blocks nor log segments found. Creating new log.
Jun 20 21:50:55 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:55.298898 9026 log.cc:824] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:55.301095 9026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6: No bootstrap required, opened a new log
I20260620 21:50:55.304505 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:55.290073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.0.193.254:41369)
0620 21:50:55.290557 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:55.290576 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:55.290619 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:50:55.291421 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:55.291447 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:55.291905 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:55.292288 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:55.293086 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:55.294609 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:55.295716 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:55.296403 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:55.296581 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:55.299441 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:55.299447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:55.299463 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:55.299489 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:55.301308 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:55.301847 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:55.301851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:55.301856 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:55.301960 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:55.302271 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:55.302273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:55.302274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:55.302701 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:55.302857 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:55.303337 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 21:50:55.303707 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":144,"threads_started":1}
I20260620 21:50:55.305768 9026 raft_consensus.cc:348] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96204bca20746faa1f5421828ec0ea6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41369 } }
I20260620 21:50:55.306078 9026 raft_consensus.cc:374] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:50:55.306145 9026 raft_consensus.cc:729] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b96204bca20746faa1f5421828ec0ea6, State: Initialized, Role: FOLLOWER
I20260620 21:50:55.306638 9026 consensus_queue.cc:260] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96204bca20746faa1f5421828ec0ea6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41369 } }
I20260620 21:50:55.306850 9026 raft_consensus.cc:388] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:55.306967 9026 raft_consensus.cc:482] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:55.307103 9026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:50:55.308216 9026 raft_consensus.cc:504] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96204bca20746faa1f5421828ec0ea6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41369 } }
I20260620 21:50:55.308871 9026 leader_election.cc:302] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b96204bca20746faa1f5421828ec0ea6; no voters:
I20260620 21:50:55.309304 9026 leader_election.cc:288] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:50:55.309433 9031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:50:55.309717 9031 raft_consensus.cc:686] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 1 LEADER]: Becoming Leader. State: Replica: b96204bca20746faa1f5421828ec0ea6, State: Running, Role: LEADER
I20260620 21:50:55.310123 9031 consensus_queue.cc:237] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96204bca20746faa1f5421828ec0ea6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41369 } }
I20260620 21:50:55.310825 9026 sys_catalog.cc:564] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:55.312283 9032 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b96204bca20746faa1f5421828ec0ea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96204bca20746faa1f5421828ec0ea6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41369 } } }
I20260620 21:50:55.312449 9032 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:55.312748 9033 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b96204bca20746faa1f5421828ec0ea6. Latest consensus state: current_term: 1 leader_uuid: "b96204bca20746faa1f5421828ec0ea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96204bca20746faa1f5421828ec0ea6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41369 } } }
I20260620 21:50:55.312829 9033 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:55.312884 9040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:55.316108 9040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:55.322291 9040 catalog_manager.cc:1347] Generated new cluster ID: 52f956bdbdb54e09bbfd02b93ff5c0fb
I20260620 21:50:55.322391 9040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:55.343806 9040 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:50:55.345182 9040 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:55.354058 9040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6: Generated new TSK 0
I20260620 21:50:55.354938 9040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:50:55.390935 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41369
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:55.504918 9052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:55.505252 9052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:55.505328 9052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:55.509552 9052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:55.509660 9052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:55.509754 9052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:55.515718 9052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:55.516899 9052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:55.517976 9052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:55.528613 9058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:55.528729 9060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:55.528632 9057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:55.528982 9052 server_base.cc:1034] running on GCE node
I20260620 21:50:55.529507 9052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:55.530265 9052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:55.531534 9052 hybrid_clock.cc:648] HybridClock initialized: now 1781992255531501 us; error 48 us; skew 500 ppm
Jun 20 21:50:55 dist-test-slave-kk5c krb5kdc[8945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:55.539354 9052 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:50:55.540350 9052 webserver.cc:460] Webserver started at http://127.0.193.193:38279/ using document root <none> and password file <none>
I20260620 21:50:55.540943 9052 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:55.541007 9052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:55.541177 9052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:55.543339 9052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "fe5cdebceaa4420d84f9d7d688064b58"
format_stamp: "Formatted at 2026-06-20 21:50:55 on dist-test-slave-kk5c"
server_key: "a2db0d9b8ee3cf25adda9b48ee35360c"
server_key_iv: "d0faea2ce4d83cb1fe0deda9d5f633de"
server_key_version: "encryptionkey@0"
I20260620 21:50:55.543882 9052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "fe5cdebceaa4420d84f9d7d688064b58"
format_stamp: "Formatted at 2026-06-20 21:50:55 on dist-test-slave-kk5c"
server_key: "a2db0d9b8ee3cf25adda9b48ee35360c"
server_key_iv: "d0faea2ce4d83cb1fe0deda9d5f633de"
server_key_version: "encryptionkey@0"
I20260620 21:50:55.547973 9052 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:50:55.550664 9067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:55.552168 9052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:55.552286 9052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "fe5cdebceaa4420d84f9d7d688064b58"
format_stamp: "Formatted at 2026-06-20 21:50:55 on dist-test-slave-kk5c"
server_key: "a2db0d9b8ee3cf25adda9b48ee35360c"
server_key_iv: "d0faea2ce4d83cb1fe0deda9d5f633de"
server_key_version: "encryptionkey@0"
I20260620 21:50:55.552390 9052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:55.569758 9052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:55.572991 9052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:55.573169 9052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:55.573733 9052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:55.574565 9052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:55.574621 9052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:55.574666 9052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:55.574682 9052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:55.585052 9052 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:36369
I20260620 21:50:55.585085 9180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:36369 every 8 connection(s)
I20260620 21:50:55.586051 9052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:55.588507 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 9052
I20260620 21:50:55.588692 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:50:55.589038 775 external_mini_cluster.cc:1428] Setting key 88f127b1a4c9e50f87f0b162c41f1c26
Jun 20 21:50:55 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
I20260620 21:50:55.602816 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:55.587983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:47373 (local address 127.0.193.254:41369)
0620 21:50:55.588165 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:55.588170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:55.589131 (+ 961us) server_negotiation.cc:407] Connection header received
0620 21:50:55.591001 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:55.591004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:55.591053 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:55.591113 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:55.592910 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:55.593862 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:55.595202 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:55.595590 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:55.595732 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:55.598614 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:55.598617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:55.598619 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:55.598646 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:55.600250 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:55.600880 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:55.600883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:55.600884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:55.600961 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:55.601273 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:55.601275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:55.601276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:55.601564 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:55.601751 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:55.602468 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 21:50:55.602626 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 21:50:55.603858 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:55.588389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:47373)
0620 21:50:55.588941 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:55.589004 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:55.590750 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:55.591345 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:55.591357 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:55.591950 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:55.592736 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:55.592755 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:55.594038 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:55.594041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:55.595028 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:55.595043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:55.595768 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:55.595771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:55.595977 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:55.596663 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:55.596680 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:55.598367 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:55.600421 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:55.600426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:55.600441 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:55.600765 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:55.601047 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:55.601050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:55.601051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:55.601189 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:55.601814 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:55.601820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:55.602162 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 21:50:55.602416 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 21:50:55.602780 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":141,"threads_started":1}
I20260620 21:50:55.605306 9181 heartbeater.cc:347] Connected to a master server at 127.0.193.254:41369
I20260620 21:50:55.605614 9181 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:55.606248 9181 heartbeater.cc:511] Master 127.0.193.254:41369 requested a full tablet report, sending...
I20260620 21:50:55.608354 8990 ts_manager.cc:194] Registered new tserver with Master: fe5cdebceaa4420d84f9d7d688064b58 (127.0.193.193:36369)
I20260620 21:50:55.610225 8990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:47373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:50:55.623790 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41369
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:55.737533 9186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:55.737926 9186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:55.738042 9186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:55.745661 9186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:55.745733 9186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:55.745796 9186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:55.752138 9186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:55.753643 9186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:55.755064 9186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:55.766023 9192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:55.766000 9191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:55.766139 9186 server_base.cc:1034] running on GCE node
W20260620 21:50:55.765990 9194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:55.766786 9186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:55.767573 9186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:55.768796 9186 hybrid_clock.cc:648] HybridClock initialized: now 1781992255768778 us; error 37 us; skew 500 ppm
Jun 20 21:50:55 dist-test-slave-kk5c krb5kdc[8945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:55.771839 9186 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:50:55.772727 9186 webserver.cc:460] Webserver started at http://127.0.193.194:39909/ using document root <none> and password file <none>
I20260620 21:50:55.773320 9186 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:55.773384 9186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:55.773607 9186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:55.775799 9186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "1598aa8117cc43ee939d01230d6fb9e7"
format_stamp: "Formatted at 2026-06-20 21:50:55 on dist-test-slave-kk5c"
server_key: "41203c0b9183df879a85d21f91db9ecf"
server_key_iv: "738e84ce14ed0b221fc0593fd12cfb69"
server_key_version: "encryptionkey@0"
I20260620 21:50:55.776382 9186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "1598aa8117cc43ee939d01230d6fb9e7"
format_stamp: "Formatted at 2026-06-20 21:50:55 on dist-test-slave-kk5c"
server_key: "41203c0b9183df879a85d21f91db9ecf"
server_key_iv: "738e84ce14ed0b221fc0593fd12cfb69"
server_key_version: "encryptionkey@0"
I20260620 21:50:55.780521 9186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 21:50:55.783299 9201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:55.784574 9186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:50:55.784705 9186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "1598aa8117cc43ee939d01230d6fb9e7"
format_stamp: "Formatted at 2026-06-20 21:50:55 on dist-test-slave-kk5c"
server_key: "41203c0b9183df879a85d21f91db9ecf"
server_key_iv: "738e84ce14ed0b221fc0593fd12cfb69"
server_key_version: "encryptionkey@0"
I20260620 21:50:55.784827 9186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:55.804761 9186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:55.808065 9186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:55.808272 9186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:55.808902 9186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:55.809832 9186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:55.809880 9186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:55.809993 9186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:55.810026 9186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:55.819725 9186 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:42531
I20260620 21:50:55.819746 9314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:42531 every 8 connection(s)
I20260620 21:50:55.820694 9186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:55.821424 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 9186
I20260620 21:50:55.821554 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:50:55.821851 775 external_mini_cluster.cc:1428] Setting key 6b0a1621bba9f5adb0aff835bbf1b4e5
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
Jun 20 21:50:55 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
I20260620 21:50:55.836228 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:55.822852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:53529 (local address 127.0.193.254:41369)
0620 21:50:55.822990 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:55.822994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:55.824203 (+ 1209us) server_negotiation.cc:407] Connection header received
0620 21:50:55.825652 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:55.825656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:55.825710 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:55.825819 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:55.827029 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:55.827650 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:55.828847 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:55.829080 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:55.829193 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:55.832067 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:55.832078 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:55.832081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:55.832107 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:55.833724 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:55.834411 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:55.834414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:55.834416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:55.834538 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:55.834923 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:55.834927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:55.834928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:55.835145 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:55.835296 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:55.835977 (+ 681us) server_negotiation.cc:299] Negotiation successful
0620 21:50:55.836102 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:50:55.837540 9317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:55.823448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:53529)
0620 21:50:55.824024 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:55.824070 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:55.825430 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:55.826009 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:55.826017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:55.826387 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:55.826869 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:55.826882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:55.827781 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:55.827785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:55.828717 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:55.828726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:55.829238 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:55.829242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:55.829417 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:55.830072 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:55.830091 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:55.831811 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:55.833905 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:55.833914 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:55.833936 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:55.834297 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:55.834646 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:55.834650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:55.834652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:55.834815 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:55.835293 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:55.835300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:55.835661 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 21:50:55.835908 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 21:50:55.836328 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":154,"threads_started":1}
I20260620 21:50:55.839066 9315 heartbeater.cc:347] Connected to a master server at 127.0.193.254:41369
I20260620 21:50:55.839392 9315 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:55.840101 9315 heartbeater.cc:511] Master 127.0.193.254:41369 requested a full tablet report, sending...
I20260620 21:50:55.841768 8990 ts_manager.cc:194] Registered new tserver with Master: 1598aa8117cc43ee939d01230d6fb9e7 (127.0.193.194:42531)
I20260620 21:50:55.842500 8990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:53529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:50:55.853988 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41369
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:55.969795 9320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:55.970112 9320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:55.970167 9320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:55.974714 9320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:55.974792 9320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:55.974887 9320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:55.980836 9320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:55.982040 9320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:55.983140 9320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:55.994520 9326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:55.994501 9328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:55.994501 9325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:55.995076 9320 server_base.cc:1034] running on GCE node
I20260620 21:50:55.995775 9320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:55.996596 9320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:55.997782 9320 hybrid_clock.cc:648] HybridClock initialized: now 1781992255997739 us; error 60 us; skew 500 ppm
Jun 20 21:50:55 dist-test-slave-kk5c krb5kdc[8945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:56.001622 9320 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:50:56.002902 9320 webserver.cc:460] Webserver started at http://127.0.193.195:39313/ using document root <none> and password file <none>
I20260620 21:50:56.003826 9320 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:56.003953 9320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:56.004215 9320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:50:56.006467 9320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "b2747356264a4eb9ac8f8fe498f84375"
format_stamp: "Formatted at 2026-06-20 21:50:56 on dist-test-slave-kk5c"
server_key: "1ba84077a3d33fe0d4485dc4607adb26"
server_key_iv: "732bda938f5ec52f6df336ec384d647a"
server_key_version: "encryptionkey@0"
I20260620 21:50:56.007073 9320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "b2747356264a4eb9ac8f8fe498f84375"
format_stamp: "Formatted at 2026-06-20 21:50:56 on dist-test-slave-kk5c"
server_key: "1ba84077a3d33fe0d4485dc4607adb26"
server_key_iv: "732bda938f5ec52f6df336ec384d647a"
server_key_version: "encryptionkey@0"
I20260620 21:50:56.011401 9320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:50:56.014866 9335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:56.017170 9320 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:50:56.017359 9320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "b2747356264a4eb9ac8f8fe498f84375"
format_stamp: "Formatted at 2026-06-20 21:50:56 on dist-test-slave-kk5c"
server_key: "1ba84077a3d33fe0d4485dc4607adb26"
server_key_iv: "732bda938f5ec52f6df336ec384d647a"
server_key_version: "encryptionkey@0"
I20260620 21:50:56.017513 9320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:56.032907 9320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:56.036523 9320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:56.036693 9320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:56.037334 9320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:56.038134 9320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:56.038182 9320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:56.038223 9320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:56.038241 9320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:56.047966 9320 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:44389
I20260620 21:50:56.047987 9448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:44389 every 8 connection(s)
I20260620 21:50:56.048996 9320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:50:56.051707 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 9320
I20260620 21:50:56.051889 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:50:56.052183 775 external_mini_cluster.cc:1428] Setting key 31826a5d89f915cafe6277ee4a50f10c
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:50:56.066491 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:56.051086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:50629 (local address 127.0.193.254:41369)
0620 21:50:56.051317 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:56.051321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:56.052913 (+ 1592us) server_negotiation.cc:407] Connection header received
0620 21:50:56.054731 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:56.054734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:56.054783 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:56.054862 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:56.056193 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.056819 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.058415 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.058957 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.059107 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:56.062038 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:56.062041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:56.062044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:56.062074 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:56.063682 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:56.064360 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:56.064364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:56.064365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:56.064448 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:56.064891 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:56.064894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:56.064895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:56.065149 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:56.065275 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:56.066168 (+ 893us) server_negotiation.cc:299] Negotiation successful
0620 21:50:56.066318 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 21:50:56.067476 9451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:56.051612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:50629)
0620 21:50:56.052722 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:56.052798 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:56.054498 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:56.055025 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:56.055034 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:56.055426 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:56.056045 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:56.056060 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.056962 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.056965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:56.058239 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:56.058251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.059270 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.059275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:56.059504 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:56.060159 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:56.060178 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:56.061847 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:50:56.063829 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:56.063835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:56.063851 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:56.064216 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:56.064560 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:50:56.064563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:50:56.064564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:50:56.064732 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:50:56.065325 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:50:56.065333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:50:56.065715 (+ 382us) client_negotiation.cc:769] Sending connection context
0620 21:50:56.065963 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 21:50:56.066331 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":162,"threads_started":1}
I20260620 21:50:56.068717 9449 heartbeater.cc:347] Connected to a master server at 127.0.193.254:41369
I20260620 21:50:56.068979 9449 heartbeater.cc:464] Registering TS with master...
I20260620 21:50:56.069481 9449 heartbeater.cc:511] Master 127.0.193.254:41369 requested a full tablet report, sending...
I20260620 21:50:56.070531 8990 ts_manager.cc:194] Registered new tserver with Master: b2747356264a4eb9ac8f8fe498f84375 (127.0.193.195:44389)
I20260620 21:50:56.071166 8990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:50629
I20260620 21:50:56.079017 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/oryx_127.0.193.193.keytab.
Entry for principal oryx/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/oryx_127.0.193.193.keytab.
WARNING: no policy specified for oryx/127.0.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/oryx_127.0.193.194.keytab.
Entry for principal oryx/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/oryx_127.0.193.194.keytab.
WARNING: no policy specified for oryx/127.0.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/oryx_127.0.193.195.keytab.
Entry for principal oryx/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/oryx_127.0.193.195.keytab.
I20260620 21:50:56.164321 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 9052
I20260620 21:50:56.170083 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 9186
I20260620 21:50:56.176120 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 9320
I20260620 21:50:56.181558 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 8958
I20260620 21:50:56.188977 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:41369
--webserver_interface=127.0.193.254
--webserver_port=37285
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:41369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:50:56.316159 9458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:56.316478 9458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:56.316520 9458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:56.321139 9458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:50:56.321229 9458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:56.321250 9458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:56.321269 9458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:50:56.321285 9458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:50:56.327050 9458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:41369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:41369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=37285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:56.328250 9458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:56.329696 9458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:56.339406 9466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:56.339390 9464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:56.339382 9463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:56.339781 9458 server_base.cc:1034] running on GCE node
I20260620 21:50:56.340390 9458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:56.341658 9458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:56.342865 9458 hybrid_clock.cc:648] HybridClock initialized: now 1781992256342848 us; error 38 us; skew 500 ppm
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:56.346295 9458 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:50:56.347434 9458 webserver.cc:460] Webserver started at http://127.0.193.254:37285/ using document root <none> and password file <none>
I20260620 21:50:56.348058 9458 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:56.348181 9458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:56.352622 9458 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 21:50:56.354959 9473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:56.356410 9458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:56.356613 9458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "b96204bca20746faa1f5421828ec0ea6"
format_stamp: "Formatted at 2026-06-20 21:50:55 on dist-test-slave-kk5c"
server_key: "b5af88cf330f2de14d11295aa67f8df7"
server_key_iv: "d514569b20bfb0ae5414884d55b89385"
server_key_version: "encryptionkey@0"
I20260620 21:50:56.357205 9458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:56.386422 9458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:56.390013 9458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:56.390321 9458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:56.398305 9458 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:41369
I20260620 21:50:56.398618 9525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:41369 every 8 connection(s)
I20260620 21:50:56.399595 9458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:50:56.404150 9526 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:50:56.405943 9526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6: Bootstrap starting.
I20260620 21:50:56.407454 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 9458
I20260620 21:50:56.407963 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:36369
--local_ip_for_outbound_sockets=127.0.193.193
--tserver_master_addrs=127.0.193.254:41369
--webserver_port=38279
--webserver_interface=127.0.193.193
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.193.193
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/oryx_127.0.193.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:50:56.423519 9526 log.cc:824] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6: Log is configured to *not* fsync() on all Append() calls
I20260620 21:50:56.430183 9526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:50:56.430593 9526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6: Bootstrap complete.
I20260620 21:50:56.434868 9526 raft_consensus.cc:348] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96204bca20746faa1f5421828ec0ea6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41369 } }
I20260620 21:50:56.435344 9526 raft_consensus.cc:729] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b96204bca20746faa1f5421828ec0ea6, State: Initialized, Role: FOLLOWER
I20260620 21:50:56.435882 9526 consensus_queue.cc:260] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96204bca20746faa1f5421828ec0ea6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41369 } }
I20260620 21:50:56.436002 9526 raft_consensus.cc:388] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:50:56.436092 9526 raft_consensus.cc:482] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:50:56.436209 9526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:50:56.438341 9526 raft_consensus.cc:504] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96204bca20746faa1f5421828ec0ea6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41369 } }
I20260620 21:50:56.438813 9526 leader_election.cc:302] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b96204bca20746faa1f5421828ec0ea6; no voters:
I20260620 21:50:56.439249 9526 leader_election.cc:288] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 21:50:56.439347 9530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:50:56.439535 9530 raft_consensus.cc:686] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [term 2 LEADER]: Becoming Leader. State: Replica: b96204bca20746faa1f5421828ec0ea6, State: Running, Role: LEADER
I20260620 21:50:56.439855 9530 consensus_queue.cc:237] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96204bca20746faa1f5421828ec0ea6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41369 } }
I20260620 21:50:56.440550 9526 sys_catalog.cc:564] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:50:56.440860 9530 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b96204bca20746faa1f5421828ec0ea6. Latest consensus state: current_term: 2 leader_uuid: "b96204bca20746faa1f5421828ec0ea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96204bca20746faa1f5421828ec0ea6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41369 } } }
I20260620 21:50:56.440985 9530 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:56.441422 9531 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b96204bca20746faa1f5421828ec0ea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b96204bca20746faa1f5421828ec0ea6" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41369 } } }
I20260620 21:50:56.441578 9531 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6 [sys.catalog]: This master's current role is: LEADER
I20260620 21:50:56.442723 9534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:50:56.445679 9534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:50:56.446261 9534 catalog_manager.cc:1259] Loaded cluster ID: 52f956bdbdb54e09bbfd02b93ff5c0fb
I20260620 21:50:56.446343 9534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:50:56.448105 9534 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:50:56.448693 9534 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b96204bca20746faa1f5421828ec0ea6: Loaded TSK: 0
I20260620 21:50:56.449679 9534 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:50:56.539698 9528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:56.540040 9528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:56.540148 9528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:56.545181 9528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:56.545285 9528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:56.545387 9528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:50:56.551066 9528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/oryx_127.0.193.193.keytab
--principal=oryx/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:36369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=38279
--tserver_master_addrs=127.0.193.254:41369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:56.552211 9528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:56.553262 9528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:50:56.563730 9528 server_base.cc:1034] running on GCE node
W20260620 21:50:56.563525 9553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:56.563856 9555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:56.563525 9552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:56.564458 9528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:56.565310 9528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:56.566525 9528 hybrid_clock.cc:648] HybridClock initialized: now 1781992256566509 us; error 44 us; skew 500 ppm
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992256, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:56.570212 9528 init.cc:377] Logged in from keytab as oryx/127.0.193.193@KRBTEST.COM (short username oryx)
I20260620 21:50:56.571472 9528 webserver.cc:460] Webserver started at http://127.0.193.193:38279/ using document root <none> and password file <none>
I20260620 21:50:56.572206 9528 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:56.572324 9528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:56.577601 9528 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:50:56.580236 9562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:56.581732 9528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:56.581869 9528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "fe5cdebceaa4420d84f9d7d688064b58"
format_stamp: "Formatted at 2026-06-20 21:50:55 on dist-test-slave-kk5c"
server_key: "a2db0d9b8ee3cf25adda9b48ee35360c"
server_key_iv: "d0faea2ce4d83cb1fe0deda9d5f633de"
server_key_version: "encryptionkey@0"
I20260620 21:50:56.582564 9528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:56.606853 9528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:56.611045 9528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:56.611335 9528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:56.611976 9528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:56.612924 9528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:56.613003 9528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:56.613066 9528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:56.613126 9528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:56.624290 9528 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:36369
I20260620 21:50:56.624315 9675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:36369 every 8 connection(s)
I20260620 21:50:56.625366 9528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:50:56.626065 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 9528
I20260620 21:50:56.626585 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:42531
--local_ip_for_outbound_sockets=127.0.193.194
--tserver_master_addrs=127.0.193.254:41369
--webserver_port=39909
--webserver_interface=127.0.193.194
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.193.194
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/oryx_127.0.193.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 21:50:56.640473 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:56.628284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:59729 (local address 127.0.193.254:41369)
0620 21:50:56.628877 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:56.628896 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:56.628943 (+ 47us) server_negotiation.cc:407] Connection header received
0620 21:50:56.630586 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:56.630597 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:56.631029 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:56.631457 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:56.632697 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.633664 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.634826 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.635370 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.635641 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:56.639692 (+ 4051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:59729: BlockingRecv error: recv got EOF from 127.0.193.193:59729 (error 108)
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":128,"threads_started":1}
I20260620 21:50:56.640488 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:56.628283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:59729)
0620 21:50:56.628799 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:56.628852 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:56.630155 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:56.631526 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:56.631535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:56.631983 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:56.632501 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:56.632516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.633812 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.633820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:56.634693 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:56.634703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.635630 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.635640 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:56.635849 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:56.636499 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:56.636516 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:56.639450 (+ 2934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":154,"threads_started":1}
W20260620 21:50:56.641078 9676 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:41369 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.193.254:41369: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:50:56.647120 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:56.641677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:39895)
0620 21:50:56.641809 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:56.641825 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:56.641927 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:56.642172 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:56.642175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:56.642353 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:56.642444 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:56.642449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.643262 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.643265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:56.643835 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:56.643842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.644295 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.644299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:56.644392 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:56.644911 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:56.644925 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:56.646922 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 21:50:56.647167 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:56.641686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:39895 (local address 127.0.193.254:41369)
0620 21:50:56.641808 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:56.641814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:56.641875 (+ 61us) server_negotiation.cc:407] Connection header received
0620 21:50:56.642013 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:56.642016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:56.642062 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:56.642159 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:56.642589 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.643115 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.643954 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.644185 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.644294 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:56.647063 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:39895: BlockingRecv error: recv got EOF from 127.0.193.193:39895 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:50:56.654090 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:56.648016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:36147)
0620 21:50:56.648164 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:56.648177 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:56.648274 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:56.648570 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:56.648574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:56.648731 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:56.648802 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:56.648807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.649760 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.649768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:56.650641 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:56.650652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.651128 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.651131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:56.651251 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:56.651792 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:56.651804 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:56.653861 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 21:50:56.654131 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:56.648195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:36147 (local address 127.0.193.254:41369)
0620 21:50:56.648344 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:56.648350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:56.648370 (+ 20us) server_negotiation.cc:407] Connection header received
0620 21:50:56.648402 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:56.648404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:56.648440 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:56.648535 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:56.648915 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.649618 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.650783 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.651012 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.651127 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:56.654017 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:36147: BlockingRecv error: recv got EOF from 127.0.193.193:36147 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 21:50:56.654942 9676 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:50:56.753618 9678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:56.753938 9678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:56.754010 9678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:56.758265 9678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:56.758373 9678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:56.758467 9678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:50:56.764514 9678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/oryx_127.0.193.194.keytab
--principal=oryx/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:42531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=39909
--tserver_master_addrs=127.0.193.254:41369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:56.765659 9678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:56.766705 9678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:56.779874 9686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:56.780241 9688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:56.780712 9685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:56.780725 9678 server_base.cc:1034] running on GCE node
I20260620 21:50:56.781273 9678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:56.782060 9678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:56.785358 9678 hybrid_clock.cc:648] HybridClock initialized: now 1781992256785315 us; error 51 us; skew 500 ppm
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992256, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:56.789469 9678 init.cc:377] Logged in from keytab as oryx/127.0.193.194@KRBTEST.COM (short username oryx)
I20260620 21:50:56.790886 9678 webserver.cc:460] Webserver started at http://127.0.193.194:39909/ using document root <none> and password file <none>
I20260620 21:50:56.791800 9678 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:56.791934 9678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:56.798354 9678 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 21:50:56.801476 9695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:56.803004 9678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:50:56.803238 9678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "1598aa8117cc43ee939d01230d6fb9e7"
format_stamp: "Formatted at 2026-06-20 21:50:55 on dist-test-slave-kk5c"
server_key: "41203c0b9183df879a85d21f91db9ecf"
server_key_iv: "738e84ce14ed0b221fc0593fd12cfb69"
server_key_version: "encryptionkey@0"
I20260620 21:50:56.803936 9678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:56.819000 9678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:56.822393 9678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:56.822638 9678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:56.823328 9678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:56.824222 9678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:56.824270 9678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:56.824329 9678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:56.824361 9678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:56.833712 9678 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:42531
I20260620 21:50:56.833730 9808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:42531 every 8 connection(s)
I20260620 21:50:56.834772 9678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:50:56.844151 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 9678
I20260620 21:50:56.844717 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:44389
--local_ip_for_outbound_sockets=127.0.193.195
--tserver_master_addrs=127.0.193.254:41369
--webserver_port=39313
--webserver_interface=127.0.193.195
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.193.195
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/oryx_127.0.193.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 21:50:56.847628 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:56.836809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:54153 (local address 127.0.193.254:41369)
0620 21:50:56.837019 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:56.837023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:56.837878 (+ 855us) server_negotiation.cc:407] Connection header received
0620 21:50:56.839505 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:56.839509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:56.839578 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:56.839683 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:56.840977 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.841561 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.842810 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.843051 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.843170 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:56.847520 (+ 4350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:54153: BlockingRecv error: recv got EOF from 127.0.193.194:54153 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 21:50:56.848435 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:56.837219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:54153)
0620 21:50:56.837698 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:56.837743 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:56.839285 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:56.839865 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:56.839872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:56.840323 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:56.840805 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:56.840819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.841689 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.841697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:56.842683 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:56.842692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.843234 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.843238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:56.843412 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:56.844074 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:56.844096 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:56.847335 (+ 3239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":112,"threads_started":1}
W20260620 21:50:56.849020 9809 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:41369 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.193.254:41369: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:50:56.855188 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:56.849525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:40721)
0620 21:50:56.849792 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:56.849807 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:56.849909 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:56.850214 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:56.850217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:56.850392 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:56.850495 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:56.850500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.851288 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.851292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:56.851860 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:56.851868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.852474 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.852477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:56.852589 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:56.853111 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:56.853123 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:56.855045 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202}
W20260620 21:50:56.855355 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:56.849665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:40721 (local address 127.0.193.254:41369)
0620 21:50:56.849792 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:56.849797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:56.849873 (+ 76us) server_negotiation.cc:407] Connection header received
0620 21:50:56.850061 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:56.850063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:56.850110 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:56.850197 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:56.850607 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.851134 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.851986 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.852259 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.852434 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:56.855221 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:40721: BlockingRecv error: recv got EOF from 127.0.193.194:40721 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:50:56.862083 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:56.855977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:40347)
0620 21:50:56.856100 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:56.856114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:56.856195 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:56.856473 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:56.856476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:56.856649 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:56.856757 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:56.856763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.857536 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.857539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:56.858400 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:56.858409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.858883 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.858887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:56.858972 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:56.859485 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:56.859498 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:56.861899 (+ 2401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 21:50:56.862116 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:56.856016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:40347 (local address 127.0.193.254:41369)
0620 21:50:56.856152 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:56.856155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:56.856168 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:50:56.856314 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:56.856317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:56.856361 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:56.856451 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:56.856928 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.857426 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.858534 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:56.858772 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:56.858870 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:56.862025 (+ 3155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:40347: BlockingRecv error: recv got EOF from 127.0.193.194:40347 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 21:50:56.862262 9809 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:50:56.965832 9812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:50:56.966135 9812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:50:56.966180 9812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:50:56.970372 9812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:50:56.970459 9812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:50:56.970526 9812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:50:56.976416 9812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/krb5kdc/oryx_127.0.193.195.keytab
--principal=oryx/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:44389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=39313
--tserver_master_addrs=127.0.193.254:41369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:50:56.977440 9812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:50:56.978384 9812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:50:56.988430 9817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:56.988529 9812 server_base.cc:1034] running on GCE node
W20260620 21:50:56.988444 9818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:50:56.988443 9820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:50:56.989290 9812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:50:56.990000 9812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:50:56.991273 9812 hybrid_clock.cc:648] HybridClock initialized: now 1781992256991250 us; error 28 us; skew 500 ppm
Jun 20 21:50:56 dist-test-slave-kk5c krb5kdc[8945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992256, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:50:56.994119 9812 init.cc:377] Logged in from keytab as oryx/127.0.193.195@KRBTEST.COM (short username oryx)
I20260620 21:50:56.995088 9812 webserver.cc:460] Webserver started at http://127.0.193.195:39313/ using document root <none> and password file <none>
I20260620 21:50:56.995693 9812 fs_manager.cc:359] Metadata directory not provided
I20260620 21:50:56.995803 9812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:50:56.999879 9812 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:57.002151 9827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:57.003508 9812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:50:57.003654 9812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "b2747356264a4eb9ac8f8fe498f84375"
format_stamp: "Formatted at 2026-06-20 21:50:56 on dist-test-slave-kk5c"
server_key: "1ba84077a3d33fe0d4485dc4607adb26"
server_key_iv: "732bda938f5ec52f6df336ec384d647a"
server_key_version: "encryptionkey@0"
I20260620 21:50:57.004217 9812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:50:57.018926 9812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:50:57.022440 9812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:50:57.022661 9812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:50:57.023355 9812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:50:57.024204 9812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:50:57.024276 9812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:57.024349 9812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:50:57.024392 9812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:50:57.034622 9812 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:44389
I20260620 21:50:57.034641 9940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:44389 every 8 connection(s)
I20260620 21:50:57.035692 9812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:50:57.042425 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 9812
Jun 20 21:50:57 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:57 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 21:50:57.048600 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:57.037604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:59901 (local address 127.0.193.254:41369)
0620 21:50:57.037766 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:57.037769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:57.038647 (+ 878us) server_negotiation.cc:407] Connection header received
0620 21:50:57.040144 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:57.040148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:57.040209 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:57.040285 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:57.041614 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.042237 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.043741 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.044040 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.044177 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:57.048457 (+ 4280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:59901: BlockingRecv error: recv got EOF from 127.0.193.195:59901 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:50:57.049477 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:57.037990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:59901)
0620 21:50:57.038468 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:57.038515 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:57.039869 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:57.040460 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:57.040468 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:57.040878 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:57.041471 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:57.041487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.042361 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.042364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:57.043590 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:57.043600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.044230 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.044233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:57.044395 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:57.045105 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:57.045123 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:57.048292 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":120,"threads_started":1}
W20260620 21:50:57.050082 9941 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:41369 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.193.254:41369: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
I20260620 21:50:57.053257 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:57.043524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.0.193.254:41369)
0620 21:50:57.043792 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:57.043796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:57.043806 (+ 10us) server_negotiation.cc:407] Connection header received
0620 21:50:57.043835 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:57.043837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:57.043873 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:57.043942 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:57.044791 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.045301 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.046462 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.046872 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.046993 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:57.048895 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:50:57.048902 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:50:57.048934 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:50:57.048966 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:50:57.051034 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:57.051515 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:57.051521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:57.051527 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:57.051643 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:50:57.051936 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:50:57.051939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:50:57.051941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:50:57.052280 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:50:57.052489 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 21:50:57.052783 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 21:50:57.053137 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
Jun 20 21:50:57 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:57 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:50:57.056177 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:57.050605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:45343)
0620 21:50:57.050789 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:57.050805 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:57.050898 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:57.051150 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:57.051153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:57.051346 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:57.051435 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:57.051441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.052181 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.052185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:57.052781 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:57.052790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.053238 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.053242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:57.053337 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:57.053894 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:57.053908 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:57.056064 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104}
W20260620 21:50:57.056488 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:57.050685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:45343 (local address 127.0.193.254:41369)
0620 21:50:57.050795 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:57.050798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:57.050849 (+ 51us) server_negotiation.cc:407] Connection header received
0620 21:50:57.050996 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:57.050998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:57.051042 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:57.051133 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:57.051581 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.052063 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.052903 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.053123 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.053248 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:57.056257 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:45343: BlockingRecv error: recv got EOF from 127.0.193.195:45343 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 21:50:57 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:57 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:50:57.061703 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:57.056815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:37351)
0620 21:50:57.056938 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:57.056953 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:57.057036 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:57.057197 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:57.057199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:57.057320 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:57.057391 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:57.057395 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.058107 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.058109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:57.058636 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:57.058643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.059123 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.059125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:57.059181 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:57.059672 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:57.059683 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:57.061603 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 21:50:57.061933 9941 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:50:57.062071 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:57.056896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:37351 (local address 127.0.193.254:41369)
0620 21:50:57.057036 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:57.057039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:57.057051 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:50:57.057100 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:57.057102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:57.057135 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:57.057189 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:57.057508 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.058011 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.058754 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.059018 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.059113 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:57.061764 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:37351: BlockingRecv error: recv got EOF from 127.0.193.195:37351 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 21:50:57 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:57 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:50:57.662205 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:57.655714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:52289)
0620 21:50:57.656044 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:57.656063 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:57.656198 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:57.656458 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:57.656460 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:57.656655 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:57.656763 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:57.656769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.657586 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.657589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:57.658215 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:57.658224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.658772 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.658774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:57.658850 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:57.659563 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:57.659576 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:57.662022 (+ 2446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
W20260620 21:50:57.662287 9947 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:57.655873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:52289 (local address 127.0.193.254:41369)
0620 21:50:57.656218 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:57.656222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:57.656236 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:50:57.656276 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:57.656278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:57.656318 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:57.656404 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:57.656887 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.657437 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.658403 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.658644 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.658737 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:57.662151 (+ 3414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:52289: BlockingRecv error: recv got EOF from 127.0.193.193:52289 (error 108)
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
Jun 20 21:50:57 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:57 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:50:57.868867 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:57.862924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:58731)
0620 21:50:57.863319 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:57.863334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:57.863437 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:57.863693 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:57.863695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:57.863881 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:57.863979 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:57.863984 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.864741 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.864744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:57.865342 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:57.865351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.865804 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.865807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:57.865873 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:57.866443 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:57.866455 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:57.868680 (+ 2225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":97,"threads_started":1}
W20260620 21:50:57.868925 9947 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:57.863045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:58731 (local address 127.0.193.254:41369)
0620 21:50:57.863282 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:57.863286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:57.863431 (+ 145us) server_negotiation.cc:407] Connection header received
0620 21:50:57.863538 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:57.863540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:57.863591 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:57.863654 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:57.864097 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.864633 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.865466 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:57.865689 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:57.865772 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:57.868812 (+ 3040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:58731: BlockingRecv error: recv got EOF from 127.0.193.194:58731 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 20 21:50:58 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:58 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:50:58.067958 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:58.062501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:35977)
0620 21:50:58.062755 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:58.062769 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:58.062862 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:58.063087 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:58.063089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:58.063332 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:58.063444 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:58.063450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:58.064290 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:58.064293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:58.064886 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:58.064894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:58.065313 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:58.065315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:58.065383 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:58.065914 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:58.065926 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:58.067832 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
W20260620 21:50:58.068128 9947 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:58.062555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:35977 (local address 127.0.193.254:41369)
0620 21:50:58.062697 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:58.062701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:58.062818 (+ 117us) server_negotiation.cc:407] Connection header received
0620 21:50:58.062938 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:58.062940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:58.062986 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:58.063049 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:58.063640 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:58.064177 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:58.065001 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:58.065211 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:58.065294 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:58.067977 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:35977: BlockingRecv error: recv got EOF from 127.0.193.195:35977 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 21:50:58 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:58 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:50:58.668797 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:58.663204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:38663)
0620 21:50:58.663540 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:58.663554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:58.663647 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:58.663946 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:58.663948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:58.664129 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:58.664233 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:58.664239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:58.664986 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:58.664989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:58.665598 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:58.665608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:58.666090 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:58.666093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:58.666158 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:58.666666 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:58.666678 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:58.668654 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":135,"threads_started":1}
W20260620 21:50:58.668958 9951 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:58.663310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:38663 (local address 127.0.193.254:41369)
0620 21:50:58.663584 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:58.663587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:58.663603 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:50:58.663738 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:58.663741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:58.663786 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:58.663870 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:58.664347 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:58.664869 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:58.665716 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:58.665935 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:58.666072 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:58.668810 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:38663: BlockingRecv error: recv got EOF from 127.0.193.193:38663 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":119,"threads_started":1}
Jun 20 21:50:58 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:58 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:50:58.875687 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:58.869786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:52995)
0620 21:50:58.870131 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:58.870146 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:58.870259 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:58.870629 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:58.870631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:58.870836 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:58.870936 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:58.870942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:58.871724 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:58.871727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:58.872379 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:58.872389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:58.872937 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:58.872941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:58.873014 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:58.873535 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:58.873553 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:58.875510 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":156,"threads_started":1}
W20260620 21:50:58.875754 9951 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:58.869913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:52995 (local address 127.0.193.254:41369)
0620 21:50:58.870125 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:58.870128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:58.870219 (+ 91us) server_negotiation.cc:407] Connection header received
0620 21:50:58.870357 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:58.870360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:58.870404 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:58.870528 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:58.871054 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:58.871609 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:58.872533 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:58.872748 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:58.872831 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:58.875643 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:52995: BlockingRecv error: recv got EOF from 127.0.193.194:52995 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Jun 20 21:50:59 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:59 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:50:59.075143 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:59.069047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:48203)
0620 21:50:59.069343 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:59.069361 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:59.069494 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:59.069858 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:59.069860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:59.070151 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:59.070258 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:59.070265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:59.071113 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:59.071116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:59.071777 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:59.071786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:59.072227 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:59.072230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:59.072305 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:59.072932 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:59.072948 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:59.074997 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
W20260620 21:50:59.075363 9951 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:59.069196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:48203 (local address 127.0.193.254:41369)
0620 21:50:59.069343 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:59.069348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:59.069427 (+ 79us) server_negotiation.cc:407] Connection header received
0620 21:50:59.069602 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:59.069607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:59.069686 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:59.069770 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:59.070351 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:59.071002 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:59.071889 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:59.072123 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:59.072204 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:59.075147 (+ 2943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:48203: BlockingRecv error: recv got EOF from 127.0.193.195:48203 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 21:50:59 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:59 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:50:59.675043 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:59.669624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:40321)
0620 21:50:59.669898 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:59.669915 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:59.670027 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:59.670253 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:59.670255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:59.670437 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:59.670548 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:59.670554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:59.671431 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:59.671434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:59.672032 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:59.672042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:59.672498 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:59.672501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:59.672560 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:59.673139 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:59.673151 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:59.674923 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
W20260620 21:50:59.675664 9955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:59.669781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:40321 (local address 127.0.193.254:41369)
0620 21:50:59.670037 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:59.670048 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:59.670064 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:50:59.670101 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:59.670103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:59.670141 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:59.670218 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:59.670727 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:59.671297 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:59.672162 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:59.672389 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:59.672491 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:59.675122 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:40321: BlockingRecv error: recv got EOF from 127.0.193.193:40321 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
Jun 20 21:50:59 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:50:59 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:50:59.882208 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:50:59.876524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:49329)
0620 21:50:59.876804 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 21:50:59.876817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:50:59.876926 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:50:59.877147 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:50:59.877149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:50:59.877324 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:50:59.877444 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:59.877450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:59.878251 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:59.878253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:59.878918 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:50:59.878928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:50:59.879522 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:50:59.879525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:50:59.879595 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:59.880225 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:50:59.880237 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:50:59.882068 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
W20260620 21:50:59.882479 9955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:50:59.876644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:49329 (local address 127.0.193.254:41369)
0620 21:50:59.876830 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 21:50:59.876834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:50:59.876925 (+ 91us) server_negotiation.cc:407] Connection header received
0620 21:50:59.876999 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:50:59.877001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:50:59.877045 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:50:59.877118 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:50:59.877608 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:59.878143 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:59.879046 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:50:59.879347 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:50:59.879486 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:50:59.882234 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:49329: BlockingRecv error: recv got EOF from 127.0.193.194:49329 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 21:51:00 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:00 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:00.082165 9957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:00.076040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:54473)
0620 21:51:00.076325 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:00.076340 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:00.076425 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:00.076676 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:00.076678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:00.076851 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:00.076983 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:00.076989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:00.078010 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:00.078014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:00.078845 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:00.078855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:00.079323 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:00.079326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:00.079389 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:00.079879 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:00.079890 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:00.082046 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":107,"threads_started":1}
W20260620 21:51:00.082278 9955 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:00.076196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:54473 (local address 127.0.193.254:41369)
0620 21:51:00.076360 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:00.076364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:00.076412 (+ 48us) server_negotiation.cc:407] Connection header received
0620 21:51:00.076522 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:00.076524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:00.076570 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:00.076639 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:00.077089 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:00.077836 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:00.078960 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:00.079173 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:00.079323 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:00.082185 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:54473: BlockingRecv error: recv got EOF from 127.0.193.195:54473 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 21:51:00 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:00 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:00.686362 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:00.676156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:48045)
0620 21:51:00.676450 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:00.676469 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:00.676584 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:00.676893 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:00.676896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:00.677118 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:00.677245 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:00.677254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:00.678236 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:00.678239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:00.678993 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:00.679004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:00.679455 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:00.679457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:00.679520 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:00.684369 (+ 4849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:00.684383 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:00.686240 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
W20260620 21:51:00.686640 9959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:00.676303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:48045 (local address 127.0.193.254:41369)
0620 21:51:00.676577 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:00.676580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:00.676594 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:51:00.676704 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:00.676708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:00.676758 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:00.676851 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:00.677381 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:00.678106 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:00.679115 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:00.679353 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:00.679439 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:00.686489 (+ 7050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:48045: BlockingRecv error: recv got EOF from 127.0.193.193:48045 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
Jun 20 21:51:00 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:00 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:00.888937 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:00.883160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:58385)
0620 21:51:00.883505 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:00.883524 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:00.883639 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:00.883929 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:00.883932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:00.884145 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:00.884242 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:00.884248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:00.885010 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:00.885016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:00.885603 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:00.885611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:00.886121 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:00.886125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:00.886207 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:00.886740 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:00.886752 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:00.888813 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":87,"threads_started":1}
W20260620 21:51:00.889165 9959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:00.883318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:58385 (local address 127.0.193.254:41369)
0620 21:51:00.883487 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:00.883492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:00.883608 (+ 116us) server_negotiation.cc:407] Connection header received
0620 21:51:00.883765 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:00.883767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:00.883813 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:00.883901 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:00.884365 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:00.884898 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:00.885729 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:00.886000 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:00.886118 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:00.889010 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:58385: BlockingRecv error: recv got EOF from 127.0.193.194:58385 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 21:51:01 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:01 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:01.089468 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:01.083066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:34227)
0620 21:51:01.083496 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:01.083512 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:01.083652 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:01.083919 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:01.083922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:01.084168 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:01.084264 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:01.084270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:01.085059 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:01.085065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:01.085650 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:01.085659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:01.086127 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:01.086130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:01.086218 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:01.086819 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:01.086833 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:01.089340 (+ 2507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":194,"threads_started":1}
W20260620 21:51:01.089749 9959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:01.083268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:34227 (local address 127.0.193.254:41369)
0620 21:51:01.083458 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:01.083463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:01.083667 (+ 204us) server_negotiation.cc:407] Connection header received
0620 21:51:01.083746 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:01.083749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:01.083807 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:01.083898 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:01.084399 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:01.084943 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:01.085776 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:01.086003 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:01.086117 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:01.089615 (+ 3498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:34227: BlockingRecv error: recv got EOF from 127.0.193.195:34227 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 21:51:01 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:01 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:01.693602 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:01.687351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:47971)
0620 21:51:01.687672 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:01.687692 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:01.687839 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:01.688133 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:01.688136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:01.688321 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:01.688413 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:01.688419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:01.689354 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:01.689357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:01.689946 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:01.689954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:01.690475 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:01.690478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:01.690550 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:01.691101 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:01.691113 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:01.693411 (+ 2298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":77,"threads_started":1}
W20260620 21:51:01.693682 9963 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:01.687448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:47971 (local address 127.0.193.254:41369)
0620 21:51:01.687832 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:01.687838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:01.687857 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:51:01.687932 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:01.687934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:01.687973 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:01.688052 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:01.688553 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:01.689218 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:01.690083 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:01.690336 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:01.690432 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:01.693550 (+ 3118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:47971: BlockingRecv error: recv got EOF from 127.0.193.193:47971 (error 108)
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":132,"threads_started":1}
Jun 20 21:51:01 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:01 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:01.895341 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:01.889835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:41429)
0620 21:51:01.890139 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:01.890153 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:01.890236 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:01.890532 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:01.890534 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:01.890725 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:01.890815 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:01.890820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:01.891625 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:01.891628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:01.892223 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:01.892231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:01.892742 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:01.892746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:01.892815 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:01.893309 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:01.893320 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:01.895154 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":101,"threads_started":1}
W20260620 21:51:01.895634 9963 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:01.889975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:41429 (local address 127.0.193.254:41369)
0620 21:51:01.890174 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:01.890177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:01.890193 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:51:01.890370 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:01.890373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:01.890422 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:01.890499 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:01.890921 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:01.891514 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:01.892356 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:01.892585 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:01.892735 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:01.895510 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:41429: BlockingRecv error: recv got EOF from 127.0.193.194:41429 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 21:51:02 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:02 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:02.097025 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:02.090417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:60105)
0620 21:51:02.090792 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:02.090809 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:02.090935 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:02.091298 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:02.091302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:02.091622 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:02.091739 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:02.091746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:02.092555 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:02.092560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:02.093250 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:02.093262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:02.093951 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:02.093955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:02.094044 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:02.094693 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:02.094708 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:02.096886 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":101,"threads_started":1}
W20260620 21:51:02.097214 9963 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:02.090582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:60105 (local address 127.0.193.254:41369)
0620 21:51:02.090792 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:02.090797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:02.090895 (+ 98us) server_negotiation.cc:407] Connection header received
0620 21:51:02.091053 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:02.091056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:02.091109 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:02.091193 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:02.091891 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:02.092437 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:02.093451 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:02.093788 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:02.093928 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:02.097055 (+ 3127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:60105: BlockingRecv error: recv got EOF from 127.0.193.195:60105 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 21:51:02 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:02 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:02.700971 9966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:02.694483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:48523)
0620 21:51:02.694860 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:02.694874 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:02.694980 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:02.695316 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:02.695319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:02.695512 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:02.695613 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:02.695619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:02.696576 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:02.696579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:02.697286 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:02.697296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:02.697833 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:02.697836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:02.697901 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:02.698534 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:02.698547 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:02.700853 (+ 2306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":148,"threads_started":1}
W20260620 21:51:02.701256 9967 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:02.694626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:48523 (local address 127.0.193.254:41369)
0620 21:51:02.694884 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:02.694888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:02.694951 (+ 63us) server_negotiation.cc:407] Connection header received
0620 21:51:02.695098 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:02.695100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:02.695149 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:02.695246 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:02.695883 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:02.696454 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:02.697407 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:02.697681 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:02.697769 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:02.701105 (+ 3336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:48523: BlockingRecv error: recv got EOF from 127.0.193.193:48523 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":59,"threads_started":1}
Jun 20 21:51:02 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:02 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:02.902312 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:02.896267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:49731)
0620 21:51:02.896582 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:02.896597 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:02.896745 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:02.897119 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:02.897121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:02.897311 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:02.897405 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:02.897410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:02.898292 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:02.898295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:02.898865 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:02.898874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:02.899580 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:02.899583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:02.899658 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:02.900217 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:02.900229 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:02.902187 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":133,"threads_started":1}
W20260620 21:51:02.902558 9967 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:02.896431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:49731 (local address 127.0.193.254:41369)
0620 21:51:02.896640 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:02.896643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:02.896695 (+ 52us) server_negotiation.cc:407] Connection header received
0620 21:51:02.896887 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:02.896889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:02.896940 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:02.897029 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:02.897596 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:02.898138 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:02.899062 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:02.899428 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:02.899532 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:02.902422 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:49731: BlockingRecv error: recv got EOF from 127.0.193.194:49731 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 21:51:03 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:03 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:03.103986 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:03.098025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:59309)
0620 21:51:03.098382 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:03.098397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:03.098531 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:03.098796 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:03.098799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:03.099033 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:03.099124 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:03.099131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:03.099937 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:03.099942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:03.100513 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:03.100520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:03.100977 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:03.100981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:03.101073 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:03.101577 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:03.101589 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:03.103789 (+ 2200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":134,"threads_started":1}
W20260620 21:51:03.104104 9967 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:03.098102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:59309 (local address 127.0.193.254:41369)
0620 21:51:03.098254 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:03.098259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:03.098482 (+ 223us) server_negotiation.cc:407] Connection header received
0620 21:51:03.098624 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:03.098628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:03.098684 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:03.098774 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:03.099286 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:03.099826 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:03.100643 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:03.100865 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:03.100974 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:03.103961 (+ 2987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:59309: BlockingRecv error: recv got EOF from 127.0.193.195:59309 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 21:51:03 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:03 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:03.708367 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:03.702097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:38301)
0620 21:51:03.702400 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:03.702414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:03.702516 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:03.702777 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:03.702779 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:03.702962 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:03.703072 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:03.703078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:03.704039 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:03.704042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:03.704670 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:03.704679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:03.705248 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:03.705251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:03.705323 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:03.705861 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:03.705874 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:03.708234 (+ 2360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
W20260620 21:51:03.708833 9971 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:03.702213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:38301 (local address 127.0.193.254:41369)
0620 21:51:03.702521 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:03.702525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:03.702542 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:51:03.702587 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:03.702590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:03.702645 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:03.702744 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:03.703289 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:03.703876 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:03.704807 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:03.705089 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:03.705182 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:03.708727 (+ 3545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:38301: BlockingRecv error: recv got EOF from 127.0.193.193:38301 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":141,"threads_started":1}
Jun 20 21:51:03 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:03 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:03.908995 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:03.903316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:44785)
0620 21:51:03.903689 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:03.903703 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:03.903838 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:03.904109 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:03.904112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:03.904327 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:03.904417 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:03.904423 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:03.905177 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:03.905183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:03.905743 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:03.905750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:03.906204 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:03.906207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:03.906289 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:03.906832 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:03.906843 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:03.908823 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":90,"threads_started":1}
W20260620 21:51:03.909078 9971 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:03.903330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:44785 (local address 127.0.193.254:41369)
0620 21:51:03.903496 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:03.903501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:03.903788 (+ 287us) server_negotiation.cc:407] Connection header received
0620 21:51:03.903944 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:03.903949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:03.904002 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:03.904088 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:03.904531 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:03.905064 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:03.905860 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:03.906086 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:03.906201 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:03.908964 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:44785: BlockingRecv error: recv got EOF from 127.0.193.194:44785 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 21:51:04 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:04 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:04.111542 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:04.104964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:36105)
0620 21:51:04.105426 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:04.105446 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:04.105615 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:04.105911 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:04.105914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:04.106159 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:04.106271 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:04.106278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:04.107069 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:04.107074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:04.107737 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:04.107746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:04.108252 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:04.108256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:04.108340 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:04.108972 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:04.108986 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:04.111421 (+ 2435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":122,"threads_started":1}
W20260620 21:51:04.111662 9971 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:04.105142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:36105 (local address 127.0.193.254:41369)
0620 21:51:04.105384 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:04.105389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:04.105548 (+ 159us) server_negotiation.cc:407] Connection header received
0620 21:51:04.105742 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:04.105745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:04.105794 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:04.105885 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:04.106409 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:04.106950 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:04.107885 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:04.108130 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:04.108249 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:04.111565 (+ 3316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:36105: BlockingRecv error: recv got EOF from 127.0.193.195:36105 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Jun 20 21:51:04 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:04 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:04.715255 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:04.709363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:55679)
0620 21:51:04.709785 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:04.709802 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:04.709966 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:04.710252 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:04.710255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:04.710462 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:04.710558 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:04.710564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:04.711474 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:04.711480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:04.712048 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:04.712056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:04.712549 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:04.712552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:04.712643 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:04.713210 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:04.713222 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:04.715049 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":156,"threads_started":1}
W20260620 21:51:04.715372 9975 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:04.709512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:55679 (local address 127.0.193.254:41369)
0620 21:51:04.709789 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:04.709797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:04.709892 (+ 95us) server_negotiation.cc:407] Connection header received
0620 21:51:04.710074 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:04.710077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:04.710131 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:04.710229 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:04.710703 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:04.711366 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:04.712177 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:04.712420 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:04.712526 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:04.715218 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:55679: BlockingRecv error: recv got EOF from 127.0.193.193:55679 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":58,"threads_started":1}
Jun 20 21:51:04 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:04 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:04.916543 9976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:04.909938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:37733)
0620 21:51:04.910299 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:04.910314 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:04.910423 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:04.910713 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:04.910715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:04.910900 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:04.911025 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:04.911030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:04.912314 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:04.912318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:04.913086 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:04.913096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:04.913547 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:04.913550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:04.913620 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:04.914276 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:04.914292 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:04.916424 (+ 2132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":123,"threads_started":1}
W20260620 21:51:04.916706 9975 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:04.910062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:37733 (local address 127.0.193.254:41369)
0620 21:51:04.910239 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:04.910244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:04.910395 (+ 151us) server_negotiation.cc:407] Connection header received
0620 21:51:04.910514 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:04.910517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:04.910572 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:04.910662 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:04.911218 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:04.912170 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:04.913201 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:04.913415 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:04.913497 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:04.916585 (+ 3088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:37733: BlockingRecv error: recv got EOF from 127.0.193.194:37733 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 21:51:05 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:05 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:05.119331 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:05.112652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:41205)
0620 21:51:05.113047 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:05.113072 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:05.113226 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:05.113537 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:05.113541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:05.113763 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:05.113862 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:05.113868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:05.114761 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:05.114767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:05.115432 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:05.115440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:05.116033 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:05.116037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:05.116163 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:05.116888 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:05.116901 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:05.119129 (+ 2228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":90,"threads_started":1}
W20260620 21:51:05.119470 9975 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:05.112798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:41205 (local address 127.0.193.254:41369)
0620 21:51:05.112983 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:05.112988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:05.113166 (+ 178us) server_negotiation.cc:407] Connection header received
0620 21:51:05.113346 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:05.113349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:05.113409 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:05.113518 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:05.114001 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:05.114627 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:05.115573 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:05.115882 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:05.115999 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:05.119342 (+ 3343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:41205: BlockingRecv error: recv got EOF from 127.0.193.195:41205 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 21:51:05 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:05 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:05.725740 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:05.716251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:50011)
0620 21:51:05.716587 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:05.716602 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:05.716695 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:05.716967 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:05.716969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:05.717162 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:05.717257 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:05.717262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:05.718082 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:05.718086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:05.718740 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:05.718749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:05.719309 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:05.719312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:05.719402 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:05.723525 (+ 4123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:05.723541 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:05.725622 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
W20260620 21:51:05.725968 9979 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:05.716387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:50011 (local address 127.0.193.254:41369)
0620 21:51:05.716661 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:05.716665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:05.716681 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:51:05.716792 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:05.716793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:05.716836 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:05.716916 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:05.717364 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:05.717892 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:05.718862 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:05.719102 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:05.719217 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:05.725811 (+ 6594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:50011: BlockingRecv error: recv got EOF from 127.0.193.193:50011 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":57,"threads_started":1}
Jun 20 21:51:05 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:05 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:05.923723 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:05.917615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:33485)
0620 21:51:05.917933 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:05.917946 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:05.918095 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:05.918386 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:05.918390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:05.918602 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:05.918698 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:05.918705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:05.919570 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:05.919574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:05.920166 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:05.920174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:05.920657 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:05.920661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:05.920750 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:05.921350 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:05.921362 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:05.923597 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":120,"threads_started":1}
W20260620 21:51:05.924089 9979 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:05.917723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:33485 (local address 127.0.193.254:41369)
0620 21:51:05.917903 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:05.917908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:05.918044 (+ 136us) server_negotiation.cc:407] Connection header received
0620 21:51:05.918208 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:05.918212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:05.918268 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:05.918360 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:05.918823 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:05.919454 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:05.920293 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:05.920537 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:05.920648 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:05.923961 (+ 3313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:33485: BlockingRecv error: recv got EOF from 127.0.193.194:33485 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 21:51:06 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:06 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:06.127626 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:06.120537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:46937)
0620 21:51:06.120824 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:06.120840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:06.120942 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:06.121330 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:06.121333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:06.121528 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:06.121679 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:06.121685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:06.122649 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:06.122652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:06.123317 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:06.123326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:06.124058 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:06.124062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:06.124279 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:06.124969 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:06.124986 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:06.127504 (+ 2518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
W20260620 21:51:06.127874 9979 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:06.120691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:46937 (local address 127.0.193.254:41369)
0620 21:51:06.120871 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:06.120874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:06.120895 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:51:06.121118 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:06.121121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:06.121173 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:06.121245 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:06.121822 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:06.122504 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:06.123609 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:06.123863 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:06.123985 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:06.127737 (+ 3752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:46937: BlockingRecv error: recv got EOF from 127.0.193.195:46937 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 21:51:06 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:06 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:06.733686 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:06.726598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:34113)
0620 21:51:06.727028 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:06.727042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:06.727133 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:06.727662 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:06.727664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:06.727849 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:06.727947 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:06.727953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:06.728923 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:06.728925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:06.729497 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:06.729506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:06.730014 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:06.730018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:06.730088 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:06.730766 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:06.730782 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:06.733459 (+ 2677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":140,"threads_started":1}
W20260620 21:51:06.734552 9983 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:06.726773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:34113 (local address 127.0.193.254:41369)
0620 21:51:06.727128 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:06.727132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:06.727148 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:51:06.727260 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:06.727263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:06.727318 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:06.727422 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:06.728239 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:06.728774 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:06.729626 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:06.729858 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:06.729957 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:06.734276 (+ 4319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:34113: BlockingRecv error: recv got EOF from 127.0.193.193:34113 (error 108)
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":66,"threads_started":1}
Jun 20 21:51:06 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:06 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:06.930927 9984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:06.924685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:37453)
0620 21:51:06.925045 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:06.925058 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:06.925188 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:06.925533 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:06.925537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:06.925761 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:06.925901 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:06.925907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:06.926688 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:06.926693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:06.927381 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:06.927391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:06.927848 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:06.927852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:06.927938 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:06.928529 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:06.928544 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:06.930675 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":153,"threads_started":1}
W20260620 21:51:06.930963 9983 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:06.924752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:37453 (local address 127.0.193.254:41369)
0620 21:51:06.924962 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:06.924966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:06.925140 (+ 174us) server_negotiation.cc:407] Connection header received
0620 21:51:06.925358 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:06.925361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:06.925410 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:06.925511 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:06.926029 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:06.926574 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:06.927502 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:06.927724 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:06.927837 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:06.930842 (+ 3005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:37453: BlockingRecv error: recv got EOF from 127.0.193.194:37453 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 21:51:07 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:07 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:07.135051 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:07.128616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:36697)
0620 21:51:07.129003 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:07.129024 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:07.129151 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:07.129497 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:07.129501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:07.129777 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:07.129947 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:07.129958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:07.130975 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:07.130977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:07.131606 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:07.131616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:07.132181 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:07.132184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:07.132256 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:07.132785 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:07.132797 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:07.134923 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":71,"threads_started":1}
W20260620 21:51:07.135251 9983 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:07.128922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:36697 (local address 127.0.193.254:41369)
0620 21:51:07.129099 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:07.129102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:07.129117 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:51:07.129282 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:07.129288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:07.129351 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:07.129434 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:07.130096 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:07.130849 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:07.131755 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:07.132020 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:07.132137 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:07.135080 (+ 2943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:36697: BlockingRecv error: recv got EOF from 127.0.193.195:36697 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 20 21:51:07 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:07 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:07.741278 9986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:07.735122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:39911)
0620 21:51:07.735442 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:07.735460 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:07.735601 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:07.735899 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:07.735902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:07.736175 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:07.736272 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:07.736278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:07.737123 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:07.737130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:07.737687 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:07.737695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:07.738160 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:07.738164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:07.738250 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:07.738822 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:07.738834 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:07.741083 (+ 2249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":109,"threads_started":1}
W20260620 21:51:07.741412 9987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:07.735243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:39911 (local address 127.0.193.254:41369)
0620 21:51:07.735483 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:07.735490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:07.735531 (+ 41us) server_negotiation.cc:407] Connection header received
0620 21:51:07.735712 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:07.735716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:07.735772 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:07.735878 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:07.736482 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:07.737008 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:07.737809 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:07.738042 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:07.738155 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:07.741233 (+ 3078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:39911: BlockingRecv error: recv got EOF from 127.0.193.193:39911 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":61,"threads_started":1}
Jun 20 21:51:07 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:07 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:07.938100 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:07.931891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:56675)
0620 21:51:07.932213 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:07.932234 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:07.932435 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:07.932701 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:07.932704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:07.932913 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:07.933005 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:07.933011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:07.933800 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:07.933807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:07.934381 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:07.934389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:07.934861 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:07.934864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:07.934943 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:07.935557 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:07.935568 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:07.937926 (+ 2358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":86,"threads_started":1}
W20260620 21:51:07.938202 9987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:07.932004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:56675 (local address 127.0.193.254:41369)
0620 21:51:07.932179 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:07.932184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:07.932365 (+ 181us) server_negotiation.cc:407] Connection header received
0620 21:51:07.932541 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:07.932545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:07.932595 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:07.932681 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:07.933123 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:07.933687 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:07.934505 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:07.934737 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:07.934858 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:07.938083 (+ 3225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:56675: BlockingRecv error: recv got EOF from 127.0.193.194:56675 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 21:51:08 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:08 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:08.142052 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:08.136075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:53209)
0620 21:51:08.136455 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:08.136474 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:08.136605 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:08.136928 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:08.136931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:08.137145 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:08.137240 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:08.137245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:08.138023 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:08.138028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:08.138612 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:08.138620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:08.139097 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:08.139100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:08.139228 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:08.139802 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:08.139814 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:08.141898 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":88,"threads_started":1}
W20260620 21:51:08.142222 9987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:08.136140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:53209 (local address 127.0.193.254:41369)
0620 21:51:08.136289 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:08.136294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:08.136551 (+ 257us) server_negotiation.cc:407] Connection header received
0620 21:51:08.136725 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:08.136727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:08.136790 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:08.136905 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:08.137366 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:08.137910 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:08.138735 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:08.138981 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:08.139093 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:08.142057 (+ 2964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:53209: BlockingRecv error: recv got EOF from 127.0.193.195:53209 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 21:51:08 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:08 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:08.748706 9990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:08.742401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:56819)
0620 21:51:08.742699 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:08.742714 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:08.742932 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:08.743371 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:08.743373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:08.743550 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:08.743644 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:08.743650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:08.744491 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:08.744495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:08.745293 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:08.745304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:08.745909 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:08.745912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:08.746004 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:08.746579 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:08.746592 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:08.748520 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":107,"threads_started":1}
W20260620 21:51:08.748780 9991 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:08.742558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:56819 (local address 127.0.193.254:41369)
0620 21:51:08.742780 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:08.742783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:08.742799 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:51:08.743082 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:08.743085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:08.743131 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:08.743314 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:08.743766 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:08.744365 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:08.745442 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:08.745782 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:08.745905 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:08.748654 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:56819: BlockingRecv error: recv got EOF from 127.0.193.193:56819 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":59,"threads_started":1}
Jun 20 21:51:08 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:08 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:08.945478 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:08.939156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:33093)
0620 21:51:08.939514 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:08.939528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:08.939649 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:08.939934 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:08.939937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:08.940118 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:08.940234 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:08.940240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:08.941044 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:08.941094 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:08.941815 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:08.941825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:08.942381 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:08.942387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:08.942475 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:08.943101 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:08.943122 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:08.945333 (+ 2211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":158,"threads_started":1}
W20260620 21:51:08.945699 9991 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:08.939371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:33093 (local address 127.0.193.254:41369)
0620 21:51:08.939509 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:08.939512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:08.939611 (+ 99us) server_negotiation.cc:407] Connection header received
0620 21:51:08.939738 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:08.939740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:08.939784 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:08.939867 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:08.940370 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:08.940924 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:08.941946 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:08.942226 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:08.942337 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:08.945506 (+ 3169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:33093: BlockingRecv error: recv got EOF from 127.0.193.194:33093 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 21:51:09 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:09 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:09.149972 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:09.142982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:45331)
0620 21:51:09.143422 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:09.143444 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:09.143612 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:09.143989 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:09.143992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:09.144231 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:09.144333 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:09.144339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:09.145195 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:09.145199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:09.146018 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:09.146027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:09.146528 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:09.146532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:09.146626 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:09.147289 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:09.147315 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:09.149689 (+ 2374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":93,"threads_started":1}
W20260620 21:51:09.150055 9991 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:09.143133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:45331 (local address 127.0.193.254:41369)
0620 21:51:09.143382 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:09.143387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:09.143541 (+ 154us) server_negotiation.cc:407] Connection header received
0620 21:51:09.143782 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:09.143785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:09.143837 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:09.143958 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:09.144467 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:09.145055 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:09.146146 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:09.146400 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:09.146525 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:09.149924 (+ 3399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:45331: BlockingRecv error: recv got EOF from 127.0.193.195:45331 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
Jun 20 21:51:09 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:09 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:09.755393 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:09.749618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:40087)
0620 21:51:09.749904 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:09.749928 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:09.750027 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:09.750267 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:09.750269 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:09.750456 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:09.750550 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:09.750556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:09.751381 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:09.751383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:09.751930 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:09.751937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:09.752419 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:09.752421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:09.752479 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:09.753067 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:09.753080 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:09.755228 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
W20260620 21:51:09.755460 9995 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:09.749757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:40087 (local address 127.0.193.254:41369)
0620 21:51:09.750007 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:09.750011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:09.750026 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:51:09.750102 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:09.750105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:09.750146 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:09.750231 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:09.750666 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:09.751286 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:09.752045 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:09.752316 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:09.752399 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:09.755358 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:40087: BlockingRecv error: recv got EOF from 127.0.193.193:40087 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
Jun 20 21:51:09 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:09 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:09.952389 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:09.946514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:58325)
0620 21:51:09.946930 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:09.946946 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:09.947080 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:09.947405 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:09.947408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:09.947593 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:09.947689 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:09.947694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:09.948649 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:09.948651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:09.949211 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:09.949219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:09.949663 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:09.949666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:09.949727 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:09.950301 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:09.950320 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:09.952268 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":88,"threads_started":1}
W20260620 21:51:09.952600 9995 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:09.946671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:58325 (local address 127.0.193.254:41369)
0620 21:51:09.946804 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:09.946808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:09.947046 (+ 238us) server_negotiation.cc:407] Connection header received
0620 21:51:09.947172 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:09.947175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:09.947267 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:09.947363 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:09.947810 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:09.948544 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:09.949323 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:09.949560 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:09.949655 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:09.952439 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:58325: BlockingRecv error: recv got EOF from 127.0.193.194:58325 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 21:51:10 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:10 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:10.157160 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:10.150952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:49969)
0620 21:51:10.151396 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:10.151415 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:10.151583 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:10.151884 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:10.151886 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:10.152195 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:10.152299 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:10.152305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:10.153222 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:10.153226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:10.153842 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:10.153850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:10.154322 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:10.154326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:10.154412 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:10.154933 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:10.154955 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:10.157036 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":89,"threads_started":1}
W20260620 21:51:10.157377 9995 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:10.151106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:49969 (local address 127.0.193.254:41369)
0620 21:51:10.151285 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:10.151290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:10.151534 (+ 244us) server_negotiation.cc:407] Connection header received
0620 21:51:10.151703 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:10.151710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:10.151768 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:10.151860 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:10.152420 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:10.153108 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:10.153973 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:10.154211 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:10.154323 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:10.157256 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:49969: BlockingRecv error: recv got EOF from 127.0.193.195:49969 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 21:51:10 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:10 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:10.766544 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:10.756300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:36125)
0620 21:51:10.756624 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:10.756640 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:10.756751 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:10.757017 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:10.757019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:10.757212 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:10.757306 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:10.757311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:10.758161 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:10.758164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:10.758817 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:10.758827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:10.759417 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:10.759420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:10.759493 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:10.764236 (+ 4743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:10.764251 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:10.766369 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":92,"threads_started":1}
W20260620 21:51:10.766690 9999 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:10.756445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:36125 (local address 127.0.193.254:41369)
0620 21:51:10.756747 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:10.756750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:10.756765 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:51:10.756828 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:10.756830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:10.756872 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:10.756950 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:10.757440 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:10.757984 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:10.758958 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:10.759253 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:10.759341 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:10.766548 (+ 7207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:36125: BlockingRecv error: recv got EOF from 127.0.193.193:36125 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":91,"threads_started":1}
Jun 20 21:51:10 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:10 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:10.960071 10000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:10.953734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:52525)
0620 21:51:10.954104 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:10.954117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:10.954251 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:10.954640 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:10.954647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:10.954899 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:10.955002 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:10.955008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:10.955922 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:10.955925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:10.956496 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:10.956503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:10.956984 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:10.956987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:10.957051 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:10.957615 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:10.957627 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:10.959861 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":100,"threads_started":1}
W20260620 21:51:10.960137 9999 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:10.953851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:52525 (local address 127.0.193.254:41369)
0620 21:51:10.954030 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:10.954035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:10.954202 (+ 167us) server_negotiation.cc:407] Connection header received
0620 21:51:10.954387 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:10.954392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:10.954470 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:10.954552 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:10.955139 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:10.955802 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:10.956632 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:10.956873 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:10.956964 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:10.960000 (+ 3036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:52525: BlockingRecv error: recv got EOF from 127.0.193.194:52525 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 21:51:11 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:11 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.195@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:11.164377 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:11.158224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.195:48859)
0620 21:51:11.158502 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:11.158516 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:11.158618 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:11.158986 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:11.158988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:11.159160 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:11.159353 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:11.159360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:11.160233 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:11.160236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:11.160830 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:11.160839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:11.161392 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:11.161396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:11.161487 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:11.162088 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:11.162100 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:11.164225 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":93,"threads_started":1}
W20260620 21:51:11.164513 9999 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:11.158334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:48859 (local address 127.0.193.254:41369)
0620 21:51:11.158503 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:11.158507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:11.158596 (+ 89us) server_negotiation.cc:407] Connection header received
0620 21:51:11.158779 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:11.158782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:11.158841 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:11.158932 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:11.159485 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:11.160066 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:11.160966 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:11.161244 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:11.161385 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:11.164397 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.195:48859: BlockingRecv error: recv got EOF from 127.0.193.195:48859 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 21:51:11 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:11 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.193@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:11.774381 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:11.767506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.193:52991)
0620 21:51:11.767974 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:11.767988 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:11.768090 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:11.768417 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:11.768419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:11.768749 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:11.768853 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:11.768859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:11.769759 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:11.769766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:11.770379 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:11.770387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:11.770929 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:11.770931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:11.770997 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:11.771782 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:11.771795 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:11.774253 (+ 2458us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":89,"threads_started":1}
W20260620 21:51:11.774674 10003 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:11.767663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:52991 (local address 127.0.193.254:41369)
0620 21:51:11.767901 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:11.767905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:11.768156 (+ 251us) server_negotiation.cc:407] Connection header received
0620 21:51:11.768208 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:11.768211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:11.768260 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:11.768350 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:11.769005 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:11.769601 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:11.770567 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:11.770801 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:11.770895 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:11.774501 (+ 3606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.193:52991: BlockingRecv error: recv got EOF from 127.0.193.193:52991 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":57,"threads_started":1}
Jun 20 21:51:11 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 21:51:11 dist-test-slave-kk5c krb5kdc[8945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.193.194@KRBTEST.COM for oryx/127.0.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 21:51:11.966626 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:11.961057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41369 (local address 127.0.193.194:60425)
0620 21:51:11.961341 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:11.961354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:11.961439 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:11.961684 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:11.961686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:11.961852 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:11.961999 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:11.962005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:11.962791 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:11.962793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:11.963416 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:11.963425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:11.963892 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:11.963895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:11.963957 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:11.964448 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:11.964458 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:11.966491 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.193.254:41369: Server oryx/127.0.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":109,"threads_started":1}
W20260620 21:51:11.966795 10003 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:11.961107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:60425 (local address 127.0.193.254:41369)
0620 21:51:11.961253 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:11.961256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:11.961420 (+ 164us) server_negotiation.cc:407] Connection header received
0620 21:51:11.961538 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:11.961540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:11.961582 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:11.961641 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:11.962098 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:11.962695 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:11.963563 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:11.963787 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:11.963867 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:11.966661 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.193.194:60425: BlockingRecv error: recv got EOF from 127.0.193.194:60425 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:51:12.043229 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 9528
I20260620 21:51:12.049083 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 9678
I20260620 21:51:12.054488 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 9812
I20260620 21:51:12.060262 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 9458
2026-06-20T21:51:12Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18173 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T21:51:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:51:12Z Disabled control of system clock
I20260620 21:51:12.092964 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:43363
--webserver_interface=127.0.193.254
--webserver_port=0
--builtin_ntp_servers=127.0.193.212:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:43363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 21:51:12.214231 10008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:51:12.214511 10008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:51:12.214546 10008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:51:12.218756 10008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:51:12.218837 10008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:51:12.218853 10008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:51:12.218870 10008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:51:12.218887 10008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:51:12.224020 10008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:43363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:43363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:51:12.225227 10008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:51:12.226773 10008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:51:12.235649 10013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:12.235682 10008 server_base.cc:1034] running on GCE node
W20260620 21:51:12.235636 10014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:12.235633 10016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:12.236593 10008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:51:12.237816 10008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:51:12.239040 10008 hybrid_clock.cc:648] HybridClock initialized: now 1781992272239015 us; error 39 us; skew 500 ppm
I20260620 21:51:12.241466 10008 webserver.cc:460] Webserver started at http://127.0.193.254:44441/ using document root <none> and password file <none>
I20260620 21:51:12.242120 10008 fs_manager.cc:359] Metadata directory not provided
I20260620 21:51:12.242224 10008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:51:12.242470 10008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:51:12.244716 10008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "68b3df4d53484d67a50617f02ff91cce"
format_stamp: "Formatted at 2026-06-20 21:51:12 on dist-test-slave-kk5c"
server_key: "eb8d42ab42ca0a2f8b502daaf7780cc6"
server_key_iv: "237d963189957846113e111ffb78b962"
server_key_version: "encryptionkey@0"
I20260620 21:51:12.245342 10008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "68b3df4d53484d67a50617f02ff91cce"
format_stamp: "Formatted at 2026-06-20 21:51:12 on dist-test-slave-kk5c"
server_key: "eb8d42ab42ca0a2f8b502daaf7780cc6"
server_key_iv: "237d963189957846113e111ffb78b962"
server_key_version: "encryptionkey@0"
I20260620 21:51:12.249588 10008 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:51:12.252596 10022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:12.254172 10008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:51:12.254361 10008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "68b3df4d53484d67a50617f02ff91cce"
format_stamp: "Formatted at 2026-06-20 21:51:12 on dist-test-slave-kk5c"
server_key: "eb8d42ab42ca0a2f8b502daaf7780cc6"
server_key_iv: "237d963189957846113e111ffb78b962"
server_key_version: "encryptionkey@0"
I20260620 21:51:12.254510 10008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:51:12.276018 10008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:51:12.276845 10008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:51:12.277103 10008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:51:12.286334 10008 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:43363
I20260620 21:51:12.286350 10074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:43363 every 8 connection(s)
I20260620 21:51:12.287424 10008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:51:12.290642 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 10008
I20260620 21:51:12.290766 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:51:12.291079 775 external_mini_cluster.cc:1428] Setting key c1a7688168e02005a17a0780dd5226ec
I20260620 21:51:12.291198 10075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:51:12.298552 10075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce: Bootstrap starting.
I20260620 21:51:12.301354 10075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce: Neither blocks nor log segments found. Creating new log.
I20260620 21:51:12.301393 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:12.292707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.0.193.254:43363)
0620 21:51:12.293451 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:12.293468 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:12.293507 (+ 39us) server_negotiation.cc:407] Connection header received
0620 21:51:12.294347 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:12.294369 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:12.294817 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:12.295252 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:12.295710 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:12.296682 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:12.297580 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:12.298084 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:12.298264 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:12.299349 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:51:12.299355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:51:12.299369 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:51:12.299811 (+ 442us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:51:12.299871 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:51:12.299894 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:51:12.300061 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:12.300260 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 21:51:12.300595 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":149,"threads_started":1}
I20260620 21:51:12.302409 10075 log.cc:824] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce: Log is configured to *not* fsync() on all Append() calls
I20260620 21:51:12.304930 10075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce: No bootstrap required, opened a new log
I20260620 21:51:12.308558 10075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b3df4d53484d67a50617f02ff91cce" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43363 } }
I20260620 21:51:12.308846 10075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:51:12.308941 10075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68b3df4d53484d67a50617f02ff91cce, State: Initialized, Role: FOLLOWER
I20260620 21:51:12.309465 10075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b3df4d53484d67a50617f02ff91cce" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43363 } }
I20260620 21:51:12.309654 10075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:51:12.309722 10075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:51:12.309819 10075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:51:12.310827 10075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b3df4d53484d67a50617f02ff91cce" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43363 } }
I20260620 21:51:12.311247 10075 leader_election.cc:302] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68b3df4d53484d67a50617f02ff91cce; no voters:
I20260620 21:51:12.311606 10075 leader_election.cc:288] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:51:12.311830 10080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:51:12.312191 10080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [term 1 LEADER]: Becoming Leader. State: Replica: 68b3df4d53484d67a50617f02ff91cce, State: Running, Role: LEADER
I20260620 21:51:12.312522 10080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b3df4d53484d67a50617f02ff91cce" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43363 } }
I20260620 21:51:12.313069 10075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:51:12.314177 10081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68b3df4d53484d67a50617f02ff91cce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b3df4d53484d67a50617f02ff91cce" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43363 } } }
I20260620 21:51:12.314304 10081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [sys.catalog]: This master's current role is: LEADER
I20260620 21:51:12.314669 10082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68b3df4d53484d67a50617f02ff91cce. Latest consensus state: current_term: 1 leader_uuid: "68b3df4d53484d67a50617f02ff91cce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b3df4d53484d67a50617f02ff91cce" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43363 } } }
I20260620 21:51:12.314731 10089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:51:12.314841 10082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce [sys.catalog]: This master's current role is: LEADER
I20260620 21:51:12.318195 10089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:51:12.324739 10089 catalog_manager.cc:1347] Generated new cluster ID: b5371fa7bc43413095877102d0fa78fa
I20260620 21:51:12.324819 10089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:51:12.340416 10089 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:51:12.341777 10089 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:51:12.347026 10089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68b3df4d53484d67a50617f02ff91cce: Generated new TSK 0
I20260620 21:51:12.347857 10089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:51:12.355829 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:43363
--builtin_ntp_servers=127.0.193.212:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 21:51:12.478184 10099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:51:12.478484 10099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:51:12.478523 10099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:51:12.482879 10099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:51:12.482946 10099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:51:12.483003 10099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:51:12.488066 10099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:43363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:51:12.489303 10099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:51:12.490741 10099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:51:12.500967 10105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:12.500967 10107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:12.500993 10104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:12.501387 10099 server_base.cc:1034] running on GCE node
I20260620 21:51:12.502087 10099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:51:12.502868 10099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:51:12.504076 10099 hybrid_clock.cc:648] HybridClock initialized: now 1781992272504025 us; error 70 us; skew 500 ppm
I20260620 21:51:12.506234 10099 webserver.cc:460] Webserver started at http://127.0.193.193:45883/ using document root <none> and password file <none>
I20260620 21:51:12.506861 10099 fs_manager.cc:359] Metadata directory not provided
I20260620 21:51:12.506959 10099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:51:12.507249 10099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:51:12.509390 10099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "9b06d9bf3eaa4eba8204f3e94c4b8c01"
format_stamp: "Formatted at 2026-06-20 21:51:12 on dist-test-slave-kk5c"
server_key: "f2cc61c98bd92871e3f9780e8721fd5d"
server_key_iv: "51159e8b480d89e110f89f811c0d44da"
server_key_version: "encryptionkey@0"
I20260620 21:51:12.509996 10099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "9b06d9bf3eaa4eba8204f3e94c4b8c01"
format_stamp: "Formatted at 2026-06-20 21:51:12 on dist-test-slave-kk5c"
server_key: "f2cc61c98bd92871e3f9780e8721fd5d"
server_key_iv: "51159e8b480d89e110f89f811c0d44da"
server_key_version: "encryptionkey@0"
I20260620 21:51:12.514266 10099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 21:51:12.517433 10113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:12.519042 10099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:51:12.519261 10099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "9b06d9bf3eaa4eba8204f3e94c4b8c01"
format_stamp: "Formatted at 2026-06-20 21:51:12 on dist-test-slave-kk5c"
server_key: "f2cc61c98bd92871e3f9780e8721fd5d"
server_key_iv: "51159e8b480d89e110f89f811c0d44da"
server_key_version: "encryptionkey@0"
I20260620 21:51:12.519416 10099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:51:12.533501 10099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:51:12.534309 10099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:51:12.534533 10099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:51:12.535295 10099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:51:12.536341 10099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:51:12.536417 10099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:12.536482 10099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:51:12.536525 10099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:12.546300 10099 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:41199
I20260620 21:51:12.546382 10226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:41199 every 8 connection(s)
I20260620 21:51:12.547421 10099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:51:12.553413 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 10099
I20260620 21:51:12.553768 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:51:12.554145 775 external_mini_cluster.cc:1428] Setting key d8e64be3a1f3025bc9d35224ad0bd777
I20260620 21:51:12.556254 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:43363
--builtin_ntp_servers=127.0.193.212:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 21:51:12.559175 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:12.549637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:53297 (local address 127.0.193.254:43363)
0620 21:51:12.549816 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:12.549822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:12.550988 (+ 1166us) server_negotiation.cc:407] Connection header received
0620 21:51:12.552771 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:12.552774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:12.552829 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:12.552915 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:12.554530 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:12.555381 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:12.556899 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:12.557192 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:12.557318 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:12.558629 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:51:12.558634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:51:12.558641 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:51:12.558656 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:51:12.558675 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:51:12.558683 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:51:12.558762 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:12.558935 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 21:51:12.559015 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 21:51:12.560320 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:12.550085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:43363 (local address 127.0.193.193:53297)
0620 21:51:12.550792 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:12.550837 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:12.552249 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:12.553179 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:12.553192 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:12.553321 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:12.554326 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:12.554349 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:12.555532 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:12.555536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:12.556743 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:12.556755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:12.557367 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:12.557371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:12.557562 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:12.558239 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:51:12.558241 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:12.558286 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:51:12.558385 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:51:12.558785 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:51:12.558794 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:51:12.558806 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 21:51:12.558937 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 21:51:12.559272 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":129,"threads_started":1}
I20260620 21:51:12.561815 10227 heartbeater.cc:347] Connected to a master server at 127.0.193.254:43363
I20260620 21:51:12.562109 10227 heartbeater.cc:464] Registering TS with master...
I20260620 21:51:12.562862 10227 heartbeater.cc:511] Master 127.0.193.254:43363 requested a full tablet report, sending...
I20260620 21:51:12.565065 10039 ts_manager.cc:194] Registered new tserver with Master: 9b06d9bf3eaa4eba8204f3e94c4b8c01 (127.0.193.193:41199)
I20260620 21:51:12.566900 10039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.193.193:53297
W20260620 21:51:12.675801 10230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:51:12.676080 10230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:51:12.676119 10230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:51:12.680255 10230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:51:12.680315 10230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:51:12.680368 10230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:51:12.685251 10230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:43363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:51:12.686336 10230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:51:12.687687 10230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:51:12.697372 10236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:12.697523 10238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:12.697402 10235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:12.697652 10230 server_base.cc:1034] running on GCE node
I20260620 21:51:12.698064 10230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:51:12.698733 10230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:51:12.699932 10230 hybrid_clock.cc:648] HybridClock initialized: now 1781992272699902 us; error 49 us; skew 500 ppm
I20260620 21:51:12.701828 10230 webserver.cc:460] Webserver started at http://127.0.193.194:34553/ using document root <none> and password file <none>
I20260620 21:51:12.702425 10230 fs_manager.cc:359] Metadata directory not provided
I20260620 21:51:12.702548 10230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:51:12.702772 10230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:51:12.704901 10230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "5afb2f10bbeb4a9ba88d64c4030414e2"
format_stamp: "Formatted at 2026-06-20 21:51:12 on dist-test-slave-kk5c"
server_key: "28718b631189e7d9d7fdc96e777c0ecc"
server_key_iv: "e855126b9fe8373263cf681b9be8416d"
server_key_version: "encryptionkey@0"
I20260620 21:51:12.705441 10230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "5afb2f10bbeb4a9ba88d64c4030414e2"
format_stamp: "Formatted at 2026-06-20 21:51:12 on dist-test-slave-kk5c"
server_key: "28718b631189e7d9d7fdc96e777c0ecc"
server_key_iv: "e855126b9fe8373263cf681b9be8416d"
server_key_version: "encryptionkey@0"
I20260620 21:51:12.709143 10230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 21:51:12.711795 10244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:12.713493 10230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:51:12.713636 10230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "5afb2f10bbeb4a9ba88d64c4030414e2"
format_stamp: "Formatted at 2026-06-20 21:51:12 on dist-test-slave-kk5c"
server_key: "28718b631189e7d9d7fdc96e777c0ecc"
server_key_iv: "e855126b9fe8373263cf681b9be8416d"
server_key_version: "encryptionkey@0"
I20260620 21:51:12.713757 10230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:51:12.728459 10230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:51:12.729144 10230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:51:12.729302 10230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:51:12.729863 10230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:51:12.730790 10230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:51:12.730859 10230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:12.730902 10230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:51:12.730917 10230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:12.740271 10230 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:39243
I20260620 21:51:12.740288 10357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:39243 every 8 connection(s)
I20260620 21:51:12.741209 10230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:51:12.743536 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 10230
I20260620 21:51:12.743660 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:51:12.743934 775 external_mini_cluster.cc:1428] Setting key 025ba1493ba3cdf3fdd7e3445d5624e6
I20260620 21:51:12.746456 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:43363
--builtin_ntp_servers=127.0.193.212:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 21:51:12.753159 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:12.743423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:54063 (local address 127.0.193.254:43363)
0620 21:51:12.743609 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:12.743615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:12.744884 (+ 1269us) server_negotiation.cc:407] Connection header received
0620 21:51:12.746674 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:12.746680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:12.746752 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:12.746847 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:12.748234 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:12.749168 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:12.750495 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:12.750921 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:12.751071 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:12.752471 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:51:12.752477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:51:12.752483 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:51:12.752504 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:51:12.752533 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:51:12.752547 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:51:12.752642 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:12.752856 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 21:51:12.752936 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 21:51:12.753945 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:12.743870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:43363 (local address 127.0.193.194:54063)
0620 21:51:12.744701 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:12.744761 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:12.746352 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:12.747087 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:12.747100 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:12.747253 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:12.748029 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:12.748058 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:12.749324 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:12.749327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:12.750328 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:12.750341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:12.751266 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:12.751271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:12.751431 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:12.752162 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:51:12.752164 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:12.752208 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:51:12.752281 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:51:12.752676 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:51:12.752683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:51:12.752689 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 21:51:12.752799 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 21:51:12.753046 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":139,"threads_started":1}
I20260620 21:51:12.755436 10358 heartbeater.cc:347] Connected to a master server at 127.0.193.254:43363
I20260620 21:51:12.755735 10358 heartbeater.cc:464] Registering TS with master...
I20260620 21:51:12.756497 10358 heartbeater.cc:511] Master 127.0.193.254:43363 requested a full tablet report, sending...
I20260620 21:51:12.757901 10039 ts_manager.cc:194] Registered new tserver with Master: 5afb2f10bbeb4a9ba88d64c4030414e2 (127.0.193.194:39243)
I20260620 21:51:12.758590 10039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.193.194:54063
W20260620 21:51:12.868469 10361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:51:12.868772 10361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:51:12.868813 10361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:51:12.873356 10361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:51:12.873437 10361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:51:12.873500 10361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:51:12.878664 10361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:43363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:51:12.879954 10361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:51:12.881328 10361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:51:12.891358 10366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:12.891358 10367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:12.891498 10361 server_base.cc:1034] running on GCE node
W20260620 21:51:12.891361 10369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:12.892459 10361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:51:12.893306 10361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:51:12.894528 10361 hybrid_clock.cc:648] HybridClock initialized: now 1781992272894512 us; error 41 us; skew 500 ppm
I20260620 21:51:12.896767 10361 webserver.cc:460] Webserver started at http://127.0.193.195:36069/ using document root <none> and password file <none>
I20260620 21:51:12.897485 10361 fs_manager.cc:359] Metadata directory not provided
I20260620 21:51:12.897583 10361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:51:12.897807 10361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:51:12.900064 10361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "4af1aac966b6432491bddd84eb0c85b6"
format_stamp: "Formatted at 2026-06-20 21:51:12 on dist-test-slave-kk5c"
server_key: "42db44ab7c000cc1f1cc115668136a71"
server_key_iv: "471fb14bd9384fb62cce0902ef8be47e"
server_key_version: "encryptionkey@0"
I20260620 21:51:12.900669 10361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "4af1aac966b6432491bddd84eb0c85b6"
format_stamp: "Formatted at 2026-06-20 21:51:12 on dist-test-slave-kk5c"
server_key: "42db44ab7c000cc1f1cc115668136a71"
server_key_iv: "471fb14bd9384fb62cce0902ef8be47e"
server_key_version: "encryptionkey@0"
I20260620 21:51:12.904661 10361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:51:12.907464 10375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:12.908744 10361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:51:12.908887 10361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "4af1aac966b6432491bddd84eb0c85b6"
format_stamp: "Formatted at 2026-06-20 21:51:12 on dist-test-slave-kk5c"
server_key: "42db44ab7c000cc1f1cc115668136a71"
server_key_iv: "471fb14bd9384fb62cce0902ef8be47e"
server_key_version: "encryptionkey@0"
I20260620 21:51:12.909060 10361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:51:12.940109 10361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:51:12.940857 10361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:51:12.941075 10361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:51:12.941707 10361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:51:12.942679 10361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:51:12.942754 10361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:12.942822 10361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:51:12.942871 10361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:12.952172 10361 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:43987
I20260620 21:51:12.952201 10488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:43987 every 8 connection(s)
I20260620 21:51:12.953115 10361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:51:12.954125 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 10361
I20260620 21:51:12.954252 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:51:12.954523 775 external_mini_cluster.cc:1428] Setting key 68f16e81562a26ebdbe63b7c4239405b
I20260620 21:51:12.963713 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:12.955251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:60461 (local address 127.0.193.254:43363)
0620 21:51:12.955405 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:12.955409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:12.957059 (+ 1650us) server_negotiation.cc:407] Connection header received
0620 21:51:12.958596 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:12.958600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:12.958656 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:12.958766 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:12.959842 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:12.960378 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:12.961557 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:12.961788 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:12.961906 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:12.963112 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:51:12.963116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:51:12.963119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:51:12.963132 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:51:12.963161 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:51:12.963170 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:51:12.963284 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:12.963443 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 21:51:12.963558 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:51:12.964663 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:12.955965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:43363 (local address 127.0.193.195:60461)
0620 21:51:12.956852 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:12.956913 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:12.958336 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:12.958943 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:12.958951 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:12.959047 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:12.959698 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:12.959714 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:12.960518 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:12.960528 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:12.961424 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:12.961433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:12.961951 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:12.961955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:12.962145 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:12.962819 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:51:12.962821 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:12.962863 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:51:12.962935 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:51:12.963284 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:51:12.963296 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:51:12.963304 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 21:51:12.963443 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 21:51:12.963743 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":159,"threads_started":1}
I20260620 21:51:12.966135 10489 heartbeater.cc:347] Connected to a master server at 127.0.193.254:43363
I20260620 21:51:12.966368 10489 heartbeater.cc:464] Registering TS with master...
I20260620 21:51:12.966889 10489 heartbeater.cc:511] Master 127.0.193.254:43363 requested a full tablet report, sending...
I20260620 21:51:12.968293 10039 ts_manager.cc:194] Registered new tserver with Master: 4af1aac966b6432491bddd84eb0c85b6 (127.0.193.195:43987)
I20260620 21:51:12.969044 10039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.193.195:60461
I20260620 21:51:12.969362 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 21:51:12.972414 10078 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:12.971581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.0.193.254:43363)
0620 21:51:12.971727 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:12.971731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:12.971906 (+ 175us) server_negotiation.cc:407] Connection header received
0620 21:51:12.972024 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:12.972026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:12.972068 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:12.972143 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:12.972312 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53826: BlockingRecv error: recv got EOF from 127.0.0.1:53826 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:51:12.972978 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 10099
I20260620 21:51:12.980229 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 10230
I20260620 21:51:12.986083 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 10361
I20260620 21:51:12.992164 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 10008
2026-06-20T21:51:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (930 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T21:51:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:51:13Z Disabled control of system clock
I20260620 21:51:13.022719 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36249
--webserver_interface=127.0.193.254
--webserver_port=0
--builtin_ntp_servers=127.0.193.212:36257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:36249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 21:51:13.141045 10500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:51:13.141350 10500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:51:13.141388 10500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:51:13.145694 10500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:51:13.145778 10500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:51:13.145794 10500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:51:13.145812 10500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:51:13.145828 10500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:51:13.150926 10500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:36249
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:51:13.152122 10500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:51:13.153497 10500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:51:13.162240 10506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:13.162232 10508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:13.162235 10505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:13.162752 10500 server_base.cc:1034] running on GCE node
I20260620 21:51:13.163427 10500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:51:13.164583 10500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:51:13.165797 10500 hybrid_clock.cc:648] HybridClock initialized: now 1781992273165781 us; error 32 us; skew 500 ppm
I20260620 21:51:13.167938 10500 webserver.cc:460] Webserver started at http://127.0.193.254:35013/ using document root <none> and password file <none>
I20260620 21:51:13.168582 10500 fs_manager.cc:359] Metadata directory not provided
I20260620 21:51:13.168692 10500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:51:13.168905 10500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:51:13.170957 10500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "9abf7f15ff46419590de3e553b5b3e5f"
format_stamp: "Formatted at 2026-06-20 21:51:13 on dist-test-slave-kk5c"
server_key: "e69a0057f391b94fbbd36d3f0267be09"
server_key_iv: "14712b807f8550088acca44fbd13dab4"
server_key_version: "encryptionkey@0"
I20260620 21:51:13.171591 10500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "9abf7f15ff46419590de3e553b5b3e5f"
format_stamp: "Formatted at 2026-06-20 21:51:13 on dist-test-slave-kk5c"
server_key: "e69a0057f391b94fbbd36d3f0267be09"
server_key_iv: "14712b807f8550088acca44fbd13dab4"
server_key_version: "encryptionkey@0"
I20260620 21:51:13.175423 10500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:51:13.178117 10514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:13.179669 10500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:51:13.179836 10500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "9abf7f15ff46419590de3e553b5b3e5f"
format_stamp: "Formatted at 2026-06-20 21:51:13 on dist-test-slave-kk5c"
server_key: "e69a0057f391b94fbbd36d3f0267be09"
server_key_iv: "14712b807f8550088acca44fbd13dab4"
server_key_version: "encryptionkey@0"
I20260620 21:51:13.179970 10500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:51:13.184594 10500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:51:13.185170 10500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:51:13.185328 10500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:51:13.192992 10500 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:36249
I20260620 21:51:13.193167 10566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:36249 every 8 connection(s)
I20260620 21:51:13.194044 10500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:51:13.197395 10567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:51:13.199390 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 10500
I20260620 21:51:13.199504 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:51:13.199802 775 external_mini_cluster.cc:1428] Setting key ccb02a7dd9bb936591f94715284d9423
I20260620 21:51:13.203620 10567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f: Bootstrap starting.
I20260620 21:51:13.205473 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:13.201294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34988 (local address 127.0.193.254:36249)
0620 21:51:13.201906 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:13.201922 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:13.201957 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:51:13.202637 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:13.202653 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:13.203008 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:13.203372 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:13.204201 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:51:13.204206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:51:13.204219 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:51:13.204229 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:51:13.204281 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:51:13.204298 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:51:13.204410 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:13.204511 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 21:51:13.204783 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":137,"threads_started":1}
I20260620 21:51:13.206097 10567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f: Neither blocks nor log segments found. Creating new log.
I20260620 21:51:13.206849 10567 log.cc:824] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:51:13.208704 10567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f: No bootstrap required, opened a new log
I20260620 21:51:13.212167 10567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abf7f15ff46419590de3e553b5b3e5f" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36249 } }
I20260620 21:51:13.212435 10567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:51:13.212509 10567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abf7f15ff46419590de3e553b5b3e5f, State: Initialized, Role: FOLLOWER
I20260620 21:51:13.212961 10567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abf7f15ff46419590de3e553b5b3e5f" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36249 } }
I20260620 21:51:13.213111 10567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:51:13.213193 10567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:51:13.213294 10567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:51:13.214217 10567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abf7f15ff46419590de3e553b5b3e5f" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36249 } }
I20260620 21:51:13.214556 10567 leader_election.cc:302] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9abf7f15ff46419590de3e553b5b3e5f; no voters:
I20260620 21:51:13.214865 10567 leader_election.cc:288] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:51:13.215025 10572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:51:13.215307 10572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [term 1 LEADER]: Becoming Leader. State: Replica: 9abf7f15ff46419590de3e553b5b3e5f, State: Running, Role: LEADER
I20260620 21:51:13.215678 10572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abf7f15ff46419590de3e553b5b3e5f" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36249 } }
I20260620 21:51:13.216210 10567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:51:13.217482 10574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9abf7f15ff46419590de3e553b5b3e5f. Latest consensus state: current_term: 1 leader_uuid: "9abf7f15ff46419590de3e553b5b3e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abf7f15ff46419590de3e553b5b3e5f" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36249 } } }
I20260620 21:51:13.217511 10573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9abf7f15ff46419590de3e553b5b3e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abf7f15ff46419590de3e553b5b3e5f" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36249 } } }
I20260620 21:51:13.217615 10573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [sys.catalog]: This master's current role is: LEADER
I20260620 21:51:13.217613 10574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f [sys.catalog]: This master's current role is: LEADER
I20260620 21:51:13.218019 10581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:51:13.221395 10581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:51:13.227592 10581 catalog_manager.cc:1347] Generated new cluster ID: 3b355e942fdc4df2822ab8f28fe771c0
I20260620 21:51:13.227687 10581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:51:13.237879 10581 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:51:13.238291 10581 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:51:13.245883 10581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9abf7f15ff46419590de3e553b5b3e5f: Generated new TSK 0
I20260620 21:51:13.246631 10581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:51:13.249624 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36249
--builtin_ntp_servers=127.0.193.212:36257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 21:51:13.369277 10591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:51:13.369560 10591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:51:13.369602 10591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:51:13.373853 10591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:51:13.373925 10591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:51:13.374022 10591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:51:13.379045 10591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:51:13.380373 10591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:51:13.381793 10591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:51:13.392190 10596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:13.392264 10599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:13.392241 10591 server_base.cc:1034] running on GCE node
W20260620 21:51:13.392190 10597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:13.392917 10591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:51:13.393657 10591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:51:13.394871 10591 hybrid_clock.cc:648] HybridClock initialized: now 1781992273394840 us; error 44 us; skew 500 ppm
I20260620 21:51:13.397060 10591 webserver.cc:460] Webserver started at http://127.0.193.193:43809/ using document root <none> and password file <none>
I20260620 21:51:13.397678 10591 fs_manager.cc:359] Metadata directory not provided
I20260620 21:51:13.397775 10591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:51:13.398020 10591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:51:13.400277 10591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "28643c6a125e4df0a457eda347034c56"
format_stamp: "Formatted at 2026-06-20 21:51:13 on dist-test-slave-kk5c"
server_key: "2fc840a2be873b0e72100da47a1d78c2"
server_key_iv: "a66dde6c64c8d46a5a30e2669ff0c26e"
server_key_version: "encryptionkey@0"
I20260620 21:51:13.400885 10591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "28643c6a125e4df0a457eda347034c56"
format_stamp: "Formatted at 2026-06-20 21:51:13 on dist-test-slave-kk5c"
server_key: "2fc840a2be873b0e72100da47a1d78c2"
server_key_iv: "a66dde6c64c8d46a5a30e2669ff0c26e"
server_key_version: "encryptionkey@0"
I20260620 21:51:13.406044 10591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 21:51:13.409829 10605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:13.411579 10591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:51:13.411747 10591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "28643c6a125e4df0a457eda347034c56"
format_stamp: "Formatted at 2026-06-20 21:51:13 on dist-test-slave-kk5c"
server_key: "2fc840a2be873b0e72100da47a1d78c2"
server_key_iv: "a66dde6c64c8d46a5a30e2669ff0c26e"
server_key_version: "encryptionkey@0"
I20260620 21:51:13.411909 10591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:51:13.416970 10591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:51:13.417683 10591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:51:13.417869 10591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:51:13.418401 10591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:51:13.419430 10591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:51:13.419479 10591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:13.419520 10591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:51:13.419536 10591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:13.429284 10591 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:37719
I20260620 21:51:13.429369 10718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:37719 every 8 connection(s)
I20260620 21:51:13.430320 10591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:51:13.436826 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 10591
I20260620 21:51:13.436981 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:51:13.437300 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:13.432532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:54849 (local address 127.0.193.254:36249)
0620 21:51:13.432692 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:13.432695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:13.433772 (+ 1077us) server_negotiation.cc:407] Connection header received
0620 21:51:13.435103 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:13.435107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:13.435155 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:13.435263 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:13.436434 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:51:13.436438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:51:13.436442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:51:13.436497 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:51:13.436523 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:51:13.436534 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:51:13.436783 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:13.437089 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 21:51:13.437151 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 21:51:13.437341 775 external_mini_cluster.cc:1428] Setting key 05e26a8894ad1124583a278e503752e8
I20260620 21:51:13.438910 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:13.432910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36249 (local address 127.0.193.193:54849)
0620 21:51:13.433613 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:13.433657 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:13.434878 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:13.435441 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:13.435448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:13.435542 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:13.436165 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:51:13.436166 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:13.436207 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:51:13.436279 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:51:13.436793 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:51:13.436805 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:51:13.436812 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:51:13.437541 (+ 729us) client_negotiation.cc:240] Negotiation successful
0620 21:51:13.437816 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":135,"threads_started":1}
I20260620 21:51:13.439690 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36249
--builtin_ntp_servers=127.0.193.212:36257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 21:51:13.440302 10719 heartbeater.cc:347] Connected to a master server at 127.0.193.254:36249
I20260620 21:51:13.440568 10719 heartbeater.cc:464] Registering TS with master...
I20260620 21:51:13.441072 10719 heartbeater.cc:511] Master 127.0.193.254:36249 requested a full tablet report, sending...
I20260620 21:51:13.442890 10531 ts_manager.cc:194] Registered new tserver with Master: 28643c6a125e4df0a457eda347034c56 (127.0.193.193:37719)
W20260620 21:51:13.553416 10722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:51:13.553701 10722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:51:13.553742 10722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:51:13.557929 10722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:51:13.557992 10722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:51:13.558082 10722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:51:13.565544 10722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:51:13.566730 10722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:51:13.568182 10722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:51:13.577847 10722 server_base.cc:1034] running on GCE node
W20260620 21:51:13.577821 10727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:13.577792 10728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:13.577809 10730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:13.578475 10722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:51:13.579233 10722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:51:13.580430 10722 hybrid_clock.cc:648] HybridClock initialized: now 1781992273580412 us; error 67 us; skew 500 ppm
I20260620 21:51:13.582240 10722 webserver.cc:460] Webserver started at http://127.0.193.194:37367/ using document root <none> and password file <none>
I20260620 21:51:13.582814 10722 fs_manager.cc:359] Metadata directory not provided
I20260620 21:51:13.582907 10722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:51:13.583135 10722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:51:13.585238 10722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "77d0b2f9153e45a28bfff875e0d5ae2a"
format_stamp: "Formatted at 2026-06-20 21:51:13 on dist-test-slave-kk5c"
server_key: "2250350f92eb6d535548e70840e4158f"
server_key_iv: "e6561d814d8d08047903ca62e252cf4f"
server_key_version: "encryptionkey@0"
I20260620 21:51:13.585807 10722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "77d0b2f9153e45a28bfff875e0d5ae2a"
format_stamp: "Formatted at 2026-06-20 21:51:13 on dist-test-slave-kk5c"
server_key: "2250350f92eb6d535548e70840e4158f"
server_key_iv: "e6561d814d8d08047903ca62e252cf4f"
server_key_version: "encryptionkey@0"
I20260620 21:51:13.589452 10722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 21:51:13.592077 10736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:13.593371 10722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:51:13.593536 10722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "77d0b2f9153e45a28bfff875e0d5ae2a"
format_stamp: "Formatted at 2026-06-20 21:51:13 on dist-test-slave-kk5c"
server_key: "2250350f92eb6d535548e70840e4158f"
server_key_iv: "e6561d814d8d08047903ca62e252cf4f"
server_key_version: "encryptionkey@0"
I20260620 21:51:13.593679 10722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:51:13.598675 10722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:51:13.599404 10722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:51:13.599607 10722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:51:13.600191 10722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:51:13.601183 10722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:51:13.601261 10722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:13.601325 10722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:51:13.601369 10722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:13.612375 10722 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:41895
I20260620 21:51:13.613214 10849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:41895 every 8 connection(s)
I20260620 21:51:13.613823 10722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:51:13.617087 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 10722
I20260620 21:51:13.617235 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:51:13.617585 775 external_mini_cluster.cc:1428] Setting key 087a1f25b8c147797f62cd226ace3fa5
I20260620 21:51:13.619887 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36249
--builtin_ntp_servers=127.0.193.212:36257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 21:51:13.621860 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:13.615985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:49645 (local address 127.0.193.254:36249)
0620 21:51:13.616181 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:13.616184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:13.617703 (+ 1519us) server_negotiation.cc:407] Connection header received
0620 21:51:13.619688 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:13.619691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:13.619733 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:13.619889 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:13.621273 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:51:13.621276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:51:13.621279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:51:13.621293 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:51:13.621316 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:51:13.621326 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:51:13.621434 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:13.621631 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 21:51:13.621682 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 21:51:13.622764 10852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:13.616529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36249 (local address 127.0.193.194:49645)
0620 21:51:13.617519 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:13.617580 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:13.619435 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:13.620051 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:13.620065 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:13.620196 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:13.620917 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:51:13.620922 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:13.620985 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:51:13.621090 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:51:13.621446 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:51:13.621458 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:51:13.621467 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 21:51:13.621653 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 21:51:13.621870 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":111,"threads_started":1}
I20260620 21:51:13.624173 10850 heartbeater.cc:347] Connected to a master server at 127.0.193.254:36249
I20260620 21:51:13.624497 10850 heartbeater.cc:464] Registering TS with master...
I20260620 21:51:13.625023 10850 heartbeater.cc:511] Master 127.0.193.254:36249 requested a full tablet report, sending...
I20260620 21:51:13.626168 10531 ts_manager.cc:194] Registered new tserver with Master: 77d0b2f9153e45a28bfff875e0d5ae2a (127.0.193.194:41895)
W20260620 21:51:13.741037 10853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:51:13.741350 10853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:51:13.741392 10853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:51:13.745733 10853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:51:13.745805 10853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:51:13.745865 10853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:51:13.751135 10853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:51:13.752351 10853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:51:13.753708 10853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:51:13.763952 10858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:13.764034 10853 server_base.cc:1034] running on GCE node
W20260620 21:51:13.763913 10859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:13.763926 10861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:13.764626 10853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:51:13.765354 10853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:51:13.766582 10853 hybrid_clock.cc:648] HybridClock initialized: now 1781992273766547 us; error 61 us; skew 500 ppm
I20260620 21:51:13.768644 10853 webserver.cc:460] Webserver started at http://127.0.193.195:45661/ using document root <none> and password file <none>
I20260620 21:51:13.769284 10853 fs_manager.cc:359] Metadata directory not provided
I20260620 21:51:13.769380 10853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:51:13.769615 10853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:51:13.771775 10853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "a7daa7fb98b74dcc80b0c6a57ea14858"
format_stamp: "Formatted at 2026-06-20 21:51:13 on dist-test-slave-kk5c"
server_key: "dfd56d5e054d96304f8ecc3314eb1ef3"
server_key_iv: "d1420d4e0d4c08a1f2256e5d43f920bb"
server_key_version: "encryptionkey@0"
I20260620 21:51:13.772356 10853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "a7daa7fb98b74dcc80b0c6a57ea14858"
format_stamp: "Formatted at 2026-06-20 21:51:13 on dist-test-slave-kk5c"
server_key: "dfd56d5e054d96304f8ecc3314eb1ef3"
server_key_iv: "d1420d4e0d4c08a1f2256e5d43f920bb"
server_key_version: "encryptionkey@0"
I20260620 21:51:13.778995 10853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 21:51:13.782389 10867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:13.784130 10853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:51:13.784297 10853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "a7daa7fb98b74dcc80b0c6a57ea14858"
format_stamp: "Formatted at 2026-06-20 21:51:13 on dist-test-slave-kk5c"
server_key: "dfd56d5e054d96304f8ecc3314eb1ef3"
server_key_iv: "d1420d4e0d4c08a1f2256e5d43f920bb"
server_key_version: "encryptionkey@0"
I20260620 21:51:13.784446 10853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:51:13.789862 10853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:51:13.790572 10853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:51:13.790800 10853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:51:13.791507 10853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:51:13.792868 10853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:51:13.792959 10853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:13.793041 10853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:51:13.793149 10853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:13.804850 10853 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:41391
I20260620 21:51:13.805924 10853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:51:13.806350 10980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:41391 every 8 connection(s)
I20260620 21:51:13.807463 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 10853
I20260620 21:51:13.807557 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:51:13.807875 775 external_mini_cluster.cc:1428] Setting key f5ff47742f67bc1a65a4e6193ec134d9
I20260620 21:51:13.813870 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:13.808565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:42985 (local address 127.0.193.254:36249)
0620 21:51:13.808767 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:13.808777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:13.810325 (+ 1548us) server_negotiation.cc:407] Connection header received
0620 21:51:13.812013 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:13.812016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:13.812054 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:13.812132 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:13.813344 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:51:13.813347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:51:13.813349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:51:13.813363 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:51:13.813391 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:51:13.813400 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:51:13.813504 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:13.813699 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 21:51:13.813748 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 21:51:13.814874 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:13.809221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36249 (local address 127.0.193.195:42985)
0620 21:51:13.810172 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:13.810230 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:13.811776 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:13.812293 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:13.812301 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:13.812410 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:13.813033 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:51:13.813035 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:13.813081 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:51:13.813156 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:51:13.813514 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:51:13.813522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:51:13.813527 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:51:13.813688 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 21:51:13.813901 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":215,"threads_started":1}
I20260620 21:51:13.816504 10981 heartbeater.cc:347] Connected to a master server at 127.0.193.254:36249
I20260620 21:51:13.816788 10981 heartbeater.cc:464] Registering TS with master...
I20260620 21:51:13.817258 10981 heartbeater.cc:511] Master 127.0.193.254:36249 requested a full tablet report, sending...
I20260620 21:51:13.818481 10531 ts_manager.cc:194] Registered new tserver with Master: a7daa7fb98b74dcc80b0c6a57ea14858 (127.0.193.195:41391)
I20260620 21:51:13.822316 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 21:51:13.825906 10570 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:51:13.824495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.0.193.254:36249)
0620 21:51:13.824644 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:13.824650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:13.825074 (+ 424us) server_negotiation.cc:407] Connection header received
0620 21:51:13.825288 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:13.825294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:13.825358 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:13.825462 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:13.825759 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35000: BlockingRecv error: recv got EOF from 127.0.0.1:35000 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:51:13.826480 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 10591
I20260620 21:51:13.834270 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 10722
I20260620 21:51:13.840158 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 10853
I20260620 21:51:13.847626 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 10500
2026-06-20T21:51:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (855 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:51:13 dist-test-slave-kk5c krb5kdc[10991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:51:13 dist-test-slave-kk5c krb5kdc[10991](info): set up 2 sockets
Jun 20 21:51:13 dist-test-slave-kk5c krb5kdc[10991](info): commencing operation
krb5kdc: starting...
W20260620 21:51:14.949025 775 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:51:15 dist-test-slave-kk5c krb5kdc[10991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:51:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:51:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:51:15.092792 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:40307
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:45605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:40307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:51:15.207828 11004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:51:15.208150 11004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:51:15.208233 11004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:51:15.212373 11004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:51:15.212493 11004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:51:15.212530 11004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:51:15.212571 11004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:51:15.212610 11004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:51:15.218223 11004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:45605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:40307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:40307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:51:15.219487 11004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:51:15.220754 11004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:51:15.230540 11009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:15.230593 11010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:15.230569 11012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:15.230858 11004 server_base.cc:1034] running on GCE node
I20260620 21:51:15.231662 11004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:51:15.232965 11004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:51:15.234196 11004 hybrid_clock.cc:648] HybridClock initialized: now 1781992275234170 us; error 45 us; skew 500 ppm
Jun 20 21:51:15 dist-test-slave-kk5c krb5kdc[10991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:51:15.237363 11004 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:51:15.238329 11004 webserver.cc:460] Webserver started at http://127.0.193.254:36491/ using document root <none> and password file <none>
I20260620 21:51:15.238945 11004 fs_manager.cc:359] Metadata directory not provided
I20260620 21:51:15.239044 11004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:51:15.239351 11004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:51:15.241523 11004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "31f1c372854e487daa4568b3eb040f8f"
format_stamp: "Formatted at 2026-06-20 21:51:15 on dist-test-slave-kk5c"
server_key: "3b38ea2e2709daa29f6b0b6822c992ea"
server_key_iv: "486c45b448f8b981b751069954e41166"
server_key_version: "encryptionkey@0"
I20260620 21:51:15.242138 11004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "31f1c372854e487daa4568b3eb040f8f"
format_stamp: "Formatted at 2026-06-20 21:51:15 on dist-test-slave-kk5c"
server_key: "3b38ea2e2709daa29f6b0b6822c992ea"
server_key_iv: "486c45b448f8b981b751069954e41166"
server_key_version: "encryptionkey@0"
I20260620 21:51:15.246656 11004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260620 21:51:15.249370 11019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:15.250692 11004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:51:15.250849 11004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "31f1c372854e487daa4568b3eb040f8f"
format_stamp: "Formatted at 2026-06-20 21:51:15 on dist-test-slave-kk5c"
server_key: "3b38ea2e2709daa29f6b0b6822c992ea"
server_key_iv: "486c45b448f8b981b751069954e41166"
server_key_version: "encryptionkey@0"
I20260620 21:51:15.250978 11004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:51:15.271821 11004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:51:15.275743 11004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:51:15.275998 11004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:51:15.284370 11004 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:40307
I20260620 21:51:15.284679 11071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:40307 every 8 connection(s)
I20260620 21:51:15.285657 11004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:51:15.288614 11072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:51:15.290162 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 11004
I20260620 21:51:15.290299 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:51:15.290627 775 external_mini_cluster.cc:1428] Setting key 1112c0040d23f088b541214208e3b8c0
I20260620 21:51:15.295336 11072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f: Bootstrap starting.
I20260620 21:51:15.297753 11072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f: Neither blocks nor log segments found. Creating new log.
I20260620 21:51:15.298525 11072 log.cc:824] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:51:15 dist-test-slave-kk5c krb5kdc[10991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:51:15.300951 11072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f: No bootstrap required, opened a new log
I20260620 21:51:15.305347 11072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f1c372854e487daa4568b3eb040f8f" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 40307 } }
I20260620 21:51:15.305642 11072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:51:15.305706 11072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f1c372854e487daa4568b3eb040f8f, State: Initialized, Role: FOLLOWER
I20260620 21:51:15.306210 11072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f1c372854e487daa4568b3eb040f8f" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 40307 } }
I20260620 21:51:15.306375 11072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:51:15.306473 11072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:51:15.306597 11072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:51:15.307067 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:15.292348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40904 (local address 127.0.193.254:40307)
0620 21:51:15.292817 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:15.292839 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:15.292881 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:51:15.293622 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:15.293641 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:15.294002 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:15.294406 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:15.295378 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:15.296554 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:15.297492 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:15.298074 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:15.298251 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:15.301480 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:51:15.301485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:51:15.301501 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:51:15.301531 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:51:15.303649 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:51:15.304196 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:51:15.304202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:51:15.304208 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:51:15.304338 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:51:15.304651 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:51:15.304656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:51:15.304658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:51:15.305116 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:51:15.305355 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:15.305661 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 21:51:15.306055 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":132,"threads_started":1}
I20260620 21:51:15.307886 11072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f1c372854e487daa4568b3eb040f8f" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 40307 } }
I20260620 21:51:15.308257 11072 leader_election.cc:302] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31f1c372854e487daa4568b3eb040f8f; no voters:
I20260620 21:51:15.308637 11072 leader_election.cc:288] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:51:15.308718 11077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:51:15.308909 11077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [term 1 LEADER]: Becoming Leader. State: Replica: 31f1c372854e487daa4568b3eb040f8f, State: Running, Role: LEADER
I20260620 21:51:15.309222 11077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f1c372854e487daa4568b3eb040f8f" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 40307 } }
I20260620 21:51:15.309963 11072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:51:15.310487 11077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31f1c372854e487daa4568b3eb040f8f. Latest consensus state: current_term: 1 leader_uuid: "31f1c372854e487daa4568b3eb040f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f1c372854e487daa4568b3eb040f8f" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 40307 } } }
I20260620 21:51:15.310604 11077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [sys.catalog]: This master's current role is: LEADER
I20260620 21:51:15.311115 11078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31f1c372854e487daa4568b3eb040f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f1c372854e487daa4568b3eb040f8f" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 40307 } } }
I20260620 21:51:15.311324 11078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f [sys.catalog]: This master's current role is: LEADER
I20260620 21:51:15.311448 11081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:51:15.314870 11081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:51:15.321206 11081 catalog_manager.cc:1347] Generated new cluster ID: b501d640925748c680dbf5a4d5f69647
I20260620 21:51:15.321303 11081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:51:15.333455 11081 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:51:15.334475 11081 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:51:15.340958 11081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f: Generated new TSK 0
I20260620 21:51:15.342183 11081 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:51:15.380196 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:40307
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:45605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:51:15.505944 11098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:51:15.506330 11098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:51:15.506423 11098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:51:15.511087 11098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:51:15.511231 11098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:51:15.511356 11098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:51:15.518988 11098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:45605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:40307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:51:15.520519 11098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:51:15.521732 11098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:51:15.532341 11103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:15.532356 11104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:15.532588 11098 server_base.cc:1034] running on GCE node
W20260620 21:51:15.532341 11106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:15.533080 11098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:51:15.534123 11098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:51:15.535355 11098 hybrid_clock.cc:648] HybridClock initialized: now 1781992275535324 us; error 50 us; skew 500 ppm
Jun 20 21:51:15 dist-test-slave-kk5c krb5kdc[10991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:51:15.538807 11098 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:51:15.539935 11098 webserver.cc:460] Webserver started at http://127.0.193.193:42889/ using document root <none> and password file <none>
I20260620 21:51:15.540737 11098 fs_manager.cc:359] Metadata directory not provided
I20260620 21:51:15.540846 11098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:51:15.541093 11098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:51:15.543460 11098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "48c20a5a9abd49569b90b177c6a48568"
format_stamp: "Formatted at 2026-06-20 21:51:15 on dist-test-slave-kk5c"
server_key: "7e37ab9d72b3277daf1c51079201d485"
server_key_iv: "ee855622535a7959de5ebe6f47c3d33e"
server_key_version: "encryptionkey@0"
I20260620 21:51:15.544090 11098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "48c20a5a9abd49569b90b177c6a48568"
format_stamp: "Formatted at 2026-06-20 21:51:15 on dist-test-slave-kk5c"
server_key: "7e37ab9d72b3277daf1c51079201d485"
server_key_iv: "ee855622535a7959de5ebe6f47c3d33e"
server_key_version: "encryptionkey@0"
I20260620 21:51:15.548357 11098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:51:15.551179 11113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:15.552816 11098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:51:15.553082 11098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "48c20a5a9abd49569b90b177c6a48568"
format_stamp: "Formatted at 2026-06-20 21:51:15 on dist-test-slave-kk5c"
server_key: "7e37ab9d72b3277daf1c51079201d485"
server_key_iv: "ee855622535a7959de5ebe6f47c3d33e"
server_key_version: "encryptionkey@0"
I20260620 21:51:15.553222 11098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:51:15.566927 11098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:51:15.570770 11098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:51:15.571054 11098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:51:15.571817 11098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:51:15.572737 11098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:51:15.572793 11098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:15.572834 11098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:51:15.572855 11098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:15.582981 11098 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:44433
I20260620 21:51:15.583001 11226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:44433 every 8 connection(s)
I20260620 21:51:15.584195 11098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:51:15.588766 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 11098
I20260620 21:51:15.588927 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:51:15.589215 775 external_mini_cluster.cc:1428] Setting key 541d81b758990d5785367b2db82bfeaf
Jun 20 21:51:15 dist-test-slave-kk5c krb5kdc[10991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
I20260620 21:51:15.599521 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:15.586303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:55695 (local address 127.0.193.254:40307)
0620 21:51:15.586456 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:15.586460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:15.587669 (+ 1209us) server_negotiation.cc:407] Connection header received
0620 21:51:15.588576 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:15.588583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:15.588644 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:15.588749 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:15.590167 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:15.590705 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:15.592005 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:15.592271 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:15.592398 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:15.595123 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:51:15.595127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:51:15.595129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:51:15.595155 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:51:15.596721 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:51:15.597490 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:51:15.597493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:51:15.597495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:51:15.597581 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:51:15.598047 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:51:15.598050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:51:15.598051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:51:15.598245 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:51:15.598325 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:15.599266 (+ 941us) server_negotiation.cc:299] Negotiation successful
0620 21:51:15.599380 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 21:51:15.600521 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:15.586717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:40307 (local address 127.0.193.193:55695)
0620 21:51:15.587510 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:15.587551 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:15.588350 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:15.589009 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:15.589017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:15.589428 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:15.590021 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:15.590038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:15.590865 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:15.590867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:15.591872 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:15.591882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:15.592439 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:15.592443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:15.592621 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:15.593340 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:15.593357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:15.594919 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:51:15.596869 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:51:15.596877 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:51:15.596900 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:51:15.597326 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:51:15.597709 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:51:15.597712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:51:15.597714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:51:15.597925 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:51:15.598365 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:51:15.598376 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:51:15.598769 (+ 393us) client_negotiation.cc:769] Sending connection context
0620 21:51:15.598998 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 21:51:15.599435 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
I20260620 21:51:15.601936 11227 heartbeater.cc:347] Connected to a master server at 127.0.193.254:40307
I20260620 21:51:15.602224 11227 heartbeater.cc:464] Registering TS with master...
I20260620 21:51:15.602866 11227 heartbeater.cc:511] Master 127.0.193.254:40307 requested a full tablet report, sending...
I20260620 21:51:15.604672 11036 ts_manager.cc:194] Registered new tserver with Master: 48c20a5a9abd49569b90b177c6a48568 (127.0.193.193:44433)
I20260620 21:51:15.606525 11036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:55695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:51:15.623116 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:40307
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:45605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:51:15.737776 11232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:51:15.738131 11232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:51:15.738178 11232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:51:15.742321 11232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:51:15.742396 11232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:51:15.742458 11232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:51:15.748021 11232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:45605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:40307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:51:15.749104 11232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:51:15.750077 11232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:51:15.759941 11240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:15.760059 11237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:15.760207 11232 server_base.cc:1034] running on GCE node
W20260620 21:51:15.759941 11238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:15.760854 11232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:51:15.761582 11232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:51:15.762791 11232 hybrid_clock.cc:648] HybridClock initialized: now 1781992275762777 us; error 33 us; skew 500 ppm
Jun 20 21:51:15 dist-test-slave-kk5c krb5kdc[10991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:51:15.765774 11232 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:51:15.766815 11232 webserver.cc:460] Webserver started at http://127.0.193.194:36773/ using document root <none> and password file <none>
I20260620 21:51:15.767535 11232 fs_manager.cc:359] Metadata directory not provided
I20260620 21:51:15.767630 11232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:51:15.767876 11232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:51:15.770051 11232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "236084e4d4c64e4b81cdd4828c5010ef"
format_stamp: "Formatted at 2026-06-20 21:51:15 on dist-test-slave-kk5c"
server_key: "5bece19717c5d3dcd93d518d293ecff3"
server_key_iv: "7cf186b454547b510706fb6e881ce6bd"
server_key_version: "encryptionkey@0"
I20260620 21:51:15.770685 11232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "236084e4d4c64e4b81cdd4828c5010ef"
format_stamp: "Formatted at 2026-06-20 21:51:15 on dist-test-slave-kk5c"
server_key: "5bece19717c5d3dcd93d518d293ecff3"
server_key_iv: "7cf186b454547b510706fb6e881ce6bd"
server_key_version: "encryptionkey@0"
I20260620 21:51:15.775476 11232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 21:51:15.778734 11247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:15.780493 11232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:51:15.780664 11232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "236084e4d4c64e4b81cdd4828c5010ef"
format_stamp: "Formatted at 2026-06-20 21:51:15 on dist-test-slave-kk5c"
server_key: "5bece19717c5d3dcd93d518d293ecff3"
server_key_iv: "7cf186b454547b510706fb6e881ce6bd"
server_key_version: "encryptionkey@0"
I20260620 21:51:15.780805 11232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:51:15.801934 11232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:51:15.806185 11232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:51:15.806447 11232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:51:15.807142 11232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:51:15.808070 11232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:51:15.808145 11232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:15.808216 11232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:51:15.808270 11232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:15.818989 11232 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:34543
I20260620 21:51:15.819016 11360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:34543 every 8 connection(s)
I20260620 21:51:15.819988 11232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:51:15.820519 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 11232
I20260620 21:51:15.820631 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:51:15.820930 775 external_mini_cluster.cc:1428] Setting key 71c6cbbd3deff9f6f3177ba70314e5d9
Jun 20 21:51:15 dist-test-slave-kk5c krb5kdc[10991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
I20260620 21:51:15.836907 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:15.821822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:44039 (local address 127.0.193.254:40307)
0620 21:51:15.821969 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:15.821974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:15.823448 (+ 1474us) server_negotiation.cc:407] Connection header received
0620 21:51:15.824402 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:15.824406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:15.824463 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:15.824543 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:15.825718 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:15.826644 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:15.828037 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:15.828470 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:15.828605 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:15.831621 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:51:15.831628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:51:15.831633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:51:15.831669 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:51:15.833523 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:51:15.834650 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:51:15.834653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:51:15.834655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:51:15.834798 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:51:15.835346 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:51:15.835352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:51:15.835357 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:51:15.835772 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:51:15.835892 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:15.836509 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 21:51:15.836673 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:51:15.838050 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:15.822319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:40307 (local address 127.0.193.194:44039)
0620 21:51:15.823283 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:15.823338 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:15.824198 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:15.824705 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:15.824712 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:15.825075 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:15.825567 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:15.825580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:15.826807 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:15.826811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:15.827891 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:15.827901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:15.828646 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:15.828649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:15.828826 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:15.829501 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:15.829518 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:15.831386 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:51:15.833721 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:51:15.833731 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:51:15.833754 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:51:15.834346 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:51:15.834967 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:51:15.834972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:51:15.834974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:51:15.835152 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:51:15.835939 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:51:15.835945 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:51:15.836225 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 21:51:15.836405 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 21:51:15.836701 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":603,"threads_started":1}
I20260620 21:51:15.839617 11361 heartbeater.cc:347] Connected to a master server at 127.0.193.254:40307
I20260620 21:51:15.839960 11361 heartbeater.cc:464] Registering TS with master...
I20260620 21:51:15.840631 11361 heartbeater.cc:511] Master 127.0.193.254:40307 requested a full tablet report, sending...
I20260620 21:51:15.841958 11036 ts_manager.cc:194] Registered new tserver with Master: 236084e4d4c64e4b81cdd4828c5010ef (127.0.193.194:34543)
I20260620 21:51:15.842572 11036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:44039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:51:15.859596 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:40307
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:45605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:51:15.983165 11366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:51:15.983521 11366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:51:15.983600 11366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:51:15.988193 11366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:51:15.988283 11366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:51:15.988379 11366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:51:15.994580 11366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:45605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:40307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:51:15.995826 11366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:51:15.997135 11366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:51:16.008044 11371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:16.008044 11372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:51:16.008075 11374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:51:16.008512 11366 server_base.cc:1034] running on GCE node
I20260620 21:51:16.009022 11366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:51:16.009874 11366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:51:16.011101 11366 hybrid_clock.cc:648] HybridClock initialized: now 1781992276011049 us; error 64 us; skew 500 ppm
Jun 20 21:51:16 dist-test-slave-kk5c krb5kdc[10991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:51:16.014490 11366 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:51:16.015566 11366 webserver.cc:460] Webserver started at http://127.0.193.195:45181/ using document root <none> and password file <none>
I20260620 21:51:16.016139 11366 fs_manager.cc:359] Metadata directory not provided
I20260620 21:51:16.016202 11366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:51:16.016430 11366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:51:16.018662 11366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "bae85aa6fec1447ab548adeb94706616"
format_stamp: "Formatted at 2026-06-20 21:51:16 on dist-test-slave-kk5c"
server_key: "1dce7a4f3cf4a5deed3276f587ba4a59"
server_key_iv: "9621e01622fe330611fded07feb43ea5"
server_key_version: "encryptionkey@0"
I20260620 21:51:16.019388 11366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "bae85aa6fec1447ab548adeb94706616"
format_stamp: "Formatted at 2026-06-20 21:51:16 on dist-test-slave-kk5c"
server_key: "1dce7a4f3cf4a5deed3276f587ba4a59"
server_key_iv: "9621e01622fe330611fded07feb43ea5"
server_key_version: "encryptionkey@0"
I20260620 21:51:16.023820 11366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:51:16.026825 11381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:16.028370 11366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:51:16.028496 11366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "bae85aa6fec1447ab548adeb94706616"
format_stamp: "Formatted at 2026-06-20 21:51:16 on dist-test-slave-kk5c"
server_key: "1dce7a4f3cf4a5deed3276f587ba4a59"
server_key_iv: "9621e01622fe330611fded07feb43ea5"
server_key_version: "encryptionkey@0"
I20260620 21:51:16.028657 11366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:51:16.048504 11366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:51:16.052001 11366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:51:16.052245 11366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:51:16.052881 11366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:51:16.053779 11366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:51:16.053855 11366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:16.053930 11366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:51:16.053978 11366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:51:16.064384 11366 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:45425
I20260620 21:51:16.064402 11494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:45425 every 8 connection(s)
I20260620 21:51:16.065404 11366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:51:16.067659 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 11366
I20260620 21:51:16.067808 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:51:16.068233 775 external_mini_cluster.cc:1428] Setting key 37e4506516de8ff4c7185cdfad906073
Jun 20 21:51:16 dist-test-slave-kk5c krb5kdc[10991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:51:16.082687 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.067555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:45157 (local address 127.0.193.254:40307)
0620 21:51:16.067739 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:16.067744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:16.068991 (+ 1247us) server_negotiation.cc:407] Connection header received
0620 21:51:16.070167 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:16.070171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:16.070227 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:16.070324 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:16.072160 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.072726 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.074223 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.074483 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.074579 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:16.077896 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:51:16.077901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:51:16.077909 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:51:16.077942 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:51:16.079895 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:51:16.080636 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:51:16.080639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:51:16.080640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:51:16.080734 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:51:16.081208 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:51:16.081212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:51:16.081214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:51:16.081516 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:51:16.081600 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:16.082348 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 21:51:16.082502 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 21:51:16.084080 11497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.067983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:40307 (local address 127.0.193.195:45157)
0620 21:51:16.068809 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:16.068872 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:16.069944 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:16.070497 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:16.070505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:16.071320 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:51:16.071945 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:16.071962 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.072890 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.072894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:16.074058 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:16.074072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.074677 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.074681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:16.074913 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:16.075693 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:51:16.075711 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:51:16.077572 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:51:16.080074 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:51:16.080081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:51:16.080097 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:51:16.080441 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:51:16.080886 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:51:16.080889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:51:16.080891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:51:16.081057 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:51:16.081668 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:51:16.081677 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:51:16.082045 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 21:51:16.082279 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 21:51:16.082697 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":140,"threads_started":1}
I20260620 21:51:16.085554 11495 heartbeater.cc:347] Connected to a master server at 127.0.193.254:40307
I20260620 21:51:16.085793 11495 heartbeater.cc:464] Registering TS with master...
I20260620 21:51:16.086385 11495 heartbeater.cc:511] Master 127.0.193.254:40307 requested a full tablet report, sending...
I20260620 21:51:16.087765 11036 ts_manager.cc:194] Registered new tserver with Master: bae85aa6fec1447ab548adeb94706616 (127.0.193.195:45425)
I20260620 21:51:16.088368 11036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:45157
I20260620 21:51:16.096378 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:51:16.108022 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.098617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.0.193.254:40307)
0620 21:51:16.098753 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:16.098756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:16.098948 (+ 192us) server_negotiation.cc:407] Connection header received
0620 21:51:16.099002 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:16.099005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:16.099063 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:16.099151 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:51:16.100083 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.100806 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.101767 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.102047 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.102143 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:16.104117 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:51:16.104121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:51:16.104123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:51:16.104152 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:51:16.105917 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:51:16.106408 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:51:16.106411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:51:16.106412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:51:16.106497 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:51:16.106858 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:51:16.106861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:51:16.106862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:51:16.107139 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:51:16.107370 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:16.107758 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 21:51:16.107882 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:51:16.112703 11036 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:51:16.116559 11036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:51:16.134119 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.127432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.0.193.194:34543)
0620 21:51:16.128275 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:16.128288 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:16.128316 (+ 28us) server_negotiation.cc:407] Connection header received
0620 21:51:16.128407 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:16.128411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:16.128632 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:16.129175 (+ 543us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:51:16.129342 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.130264 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.132363 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.133224 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.133429 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:16.133608 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:16.133818 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 21:51:16.133940 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":105,"threads_started":1}
I20260620 21:51:16.134106 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.127161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:34543 (local address 127.0.0.1:56956)
0620 21:51:16.127721 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:16.127774 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:16.127905 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:16.128815 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:16.128821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:16.128843 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:51:16.129012 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:16.129019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.130641 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.130648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:16.132185 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:16.132197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.133441 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.133445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:16.133561 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:16.133599 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 21:51:16.133779 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 21:51:16.133944 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":91,"threads_started":1}
I20260620 21:51:16.134390 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.128299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.0.193.193:44433)
0620 21:51:16.128690 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:16.128696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:16.128713 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:51:16.128773 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:16.128777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:16.128916 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:16.129636 (+ 720us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:51:16.129772 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.131079 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.132571 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.133431 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.133937 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:16.134087 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:16.134138 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 21:51:16.134233 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":130,"threads_started":1}
I20260620 21:51:16.134549 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.127906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:44433 (local address 127.0.0.1:57014)
0620 21:51:16.128267 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:16.128280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:16.128367 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:16.129079 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:16.129081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:16.129113 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:51:16.129227 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:16.129234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.131231 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.131235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:16.132441 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:16.132453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.133664 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.133667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:16.133759 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:16.133781 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:51:16.133833 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 21:51:16.134410 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":538,"spinlock_wait_cycles":8832,"thread_start_us":105,"threads_started":1}
I20260620 21:51:16.134593 11508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.127479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:45425 (local address 127.0.0.1:49358)
0620 21:51:16.127896 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:16.127913 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:16.128009 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:16.129079 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:16.129081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:16.129108 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:51:16.129438 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:16.129447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.131177 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.131182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:16.132591 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:16.132600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.133843 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.133846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:16.133954 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:16.133973 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 21:51:16.134038 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 21:51:16.134105 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":235,"spinlock_wait_cycles":896,"thread_start_us":104,"threads_started":1}
I20260620 21:51:16.135006 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.127981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.0.193.195:45425)
0620 21:51:16.128622 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:16.128634 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:16.128658 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:51:16.128733 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:16.128739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:16.128910 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:16.129179 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:51:16.129562 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.130821 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.132716 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.133535 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.133641 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:16.133782 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:16.134177 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 21:51:16.134848 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":167,"threads_started":1}
I20260620 21:51:16.137622 11429 tablet_service.cc:1467] Processing CreateTablet for tablet e7adffe32baa4039a4ce3b9ddf4a3562 (DEFAULT_TABLE table=test-table [id=4ab59bab677043df8ce2b4bee94f49db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:51:16.137831 11161 tablet_service.cc:1467] Processing CreateTablet for tablet e7adffe32baa4039a4ce3b9ddf4a3562 (DEFAULT_TABLE table=test-table [id=4ab59bab677043df8ce2b4bee94f49db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:51:16.137831 11295 tablet_service.cc:1467] Processing CreateTablet for tablet e7adffe32baa4039a4ce3b9ddf4a3562 (DEFAULT_TABLE table=test-table [id=4ab59bab677043df8ce2b4bee94f49db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:51:16.138594 11429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7adffe32baa4039a4ce3b9ddf4a3562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:51:16.138917 11295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7adffe32baa4039a4ce3b9ddf4a3562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:51:16.138983 11161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7adffe32baa4039a4ce3b9ddf4a3562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:51:16.145568 11512 tablet_bootstrap.cc:492] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616: Bootstrap starting.
I20260620 21:51:16.147531 11514 tablet_bootstrap.cc:492] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef: Bootstrap starting.
I20260620 21:51:16.147537 11513 tablet_bootstrap.cc:492] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568: Bootstrap starting.
I20260620 21:51:16.148578 11512 tablet_bootstrap.cc:654] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616: Neither blocks nor log segments found. Creating new log.
I20260620 21:51:16.149773 11512 log.cc:824] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616: Log is configured to *not* fsync() on all Append() calls
I20260620 21:51:16.150070 11513 tablet_bootstrap.cc:654] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568: Neither blocks nor log segments found. Creating new log.
I20260620 21:51:16.150588 11514 tablet_bootstrap.cc:654] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef: Neither blocks nor log segments found. Creating new log.
I20260620 21:51:16.151070 11513 log.cc:824] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568: Log is configured to *not* fsync() on all Append() calls
I20260620 21:51:16.151505 11514 log.cc:824] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef: Log is configured to *not* fsync() on all Append() calls
I20260620 21:51:16.152133 11512 tablet_bootstrap.cc:492] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616: No bootstrap required, opened a new log
I20260620 21:51:16.152354 11512 ts_tablet_manager.cc:1397] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 21:51:16.153107 11513 tablet_bootstrap.cc:492] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568: No bootstrap required, opened a new log
I20260620 21:51:16.153340 11513 ts_tablet_manager.cc:1397] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 21:51:16.155757 11512 raft_consensus.cc:348] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "236084e4d4c64e4b81cdd4828c5010ef" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34543 } } peers { permanent_uuid: "bae85aa6fec1447ab548adeb94706616" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45425 } } peers { permanent_uuid: "48c20a5a9abd49569b90b177c6a48568" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 44433 } }
I20260620 21:51:16.156090 11512 raft_consensus.cc:374] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:51:16.156147 11512 raft_consensus.cc:729] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae85aa6fec1447ab548adeb94706616, State: Initialized, Role: FOLLOWER
I20260620 21:51:16.156688 11512 consensus_queue.cc:260] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "236084e4d4c64e4b81cdd4828c5010ef" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34543 } } peers { permanent_uuid: "bae85aa6fec1447ab548adeb94706616" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45425 } } peers { permanent_uuid: "48c20a5a9abd49569b90b177c6a48568" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 44433 } }
I20260620 21:51:16.157501 11495 heartbeater.cc:503] Master 127.0.193.254:40307 was elected leader, sending a full tablet report...
I20260620 21:51:16.157784 11512 ts_tablet_manager.cc:1428] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 21:51:16.158269 11514 tablet_bootstrap.cc:492] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef: No bootstrap required, opened a new log
I20260620 21:51:16.158186 11513 raft_consensus.cc:348] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "236084e4d4c64e4b81cdd4828c5010ef" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34543 } } peers { permanent_uuid: "bae85aa6fec1447ab548adeb94706616" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45425 } } peers { permanent_uuid: "48c20a5a9abd49569b90b177c6a48568" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 44433 } }
I20260620 21:51:16.158480 11513 raft_consensus.cc:374] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:51:16.158530 11514 ts_tablet_manager.cc:1397] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260620 21:51:16.158557 11513 raft_consensus.cc:729] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c20a5a9abd49569b90b177c6a48568, State: Initialized, Role: FOLLOWER
I20260620 21:51:16.159056 11513 consensus_queue.cc:260] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "236084e4d4c64e4b81cdd4828c5010ef" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34543 } } peers { permanent_uuid: "bae85aa6fec1447ab548adeb94706616" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45425 } } peers { permanent_uuid: "48c20a5a9abd49569b90b177c6a48568" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 44433 } }
I20260620 21:51:16.159818 11227 heartbeater.cc:503] Master 127.0.193.254:40307 was elected leader, sending a full tablet report...
I20260620 21:51:16.160187 11513 ts_tablet_manager.cc:1428] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 21:51:16.162739 11514 raft_consensus.cc:348] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "236084e4d4c64e4b81cdd4828c5010ef" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34543 } } peers { permanent_uuid: "bae85aa6fec1447ab548adeb94706616" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45425 } } peers { permanent_uuid: "48c20a5a9abd49569b90b177c6a48568" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 44433 } }
I20260620 21:51:16.163084 11514 raft_consensus.cc:374] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:51:16.163161 11514 raft_consensus.cc:729] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 236084e4d4c64e4b81cdd4828c5010ef, State: Initialized, Role: FOLLOWER
I20260620 21:51:16.163669 11514 consensus_queue.cc:260] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "236084e4d4c64e4b81cdd4828c5010ef" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34543 } } peers { permanent_uuid: "bae85aa6fec1447ab548adeb94706616" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45425 } } peers { permanent_uuid: "48c20a5a9abd49569b90b177c6a48568" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 44433 } }
I20260620 21:51:16.164364 11361 heartbeater.cc:503] Master 127.0.193.254:40307 was elected leader, sending a full tablet report...
I20260620 21:51:16.164711 11514 ts_tablet_manager.cc:1428] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:51:16.269708 11518 raft_consensus.cc:482] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:51:16.269987 11518 raft_consensus.cc:504] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "236084e4d4c64e4b81cdd4828c5010ef" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34543 } } peers { permanent_uuid: "bae85aa6fec1447ab548adeb94706616" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45425 } } peers { permanent_uuid: "48c20a5a9abd49569b90b177c6a48568" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 44433 } }
I20260620 21:51:16.271188 11518 leader_election.cc:288] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 236084e4d4c64e4b81cdd4828c5010ef (127.0.193.194:34543), 48c20a5a9abd49569b90b177c6a48568 (127.0.193.193:44433)
I20260620 21:51:16.276815 11497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.271492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:34543 (local address 127.0.193.195:59977)
0620 21:51:16.271847 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:16.271869 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:16.272006 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:16.272453 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:16.272456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:16.272483 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:51:16.272617 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:16.272627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.273714 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.273717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:16.275293 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:16.275304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.276278 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.276282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:16.276433 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:16.276465 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 21:51:16.276561 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 21:51:16.276647 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":16}
I20260620 21:51:16.277000 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.271617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:59977 (local address 127.0.193.194:34543)
0620 21:51:16.271830 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:16.271836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:16.271974 (+ 138us) server_negotiation.cc:407] Connection header received
0620 21:51:16.272155 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:16.272158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:16.272215 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:16.272368 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:51:16.272760 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.273387 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.275506 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.276144 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.276263 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:16.276310 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:16.276623 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 21:51:16.276811 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 21:51:16.277333 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.272012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:55191 (local address 127.0.193.193:44433)
0620 21:51:16.272139 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:16.272144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:16.272430 (+ 286us) server_negotiation.cc:407] Connection header received
0620 21:51:16.272493 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:16.272496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:16.272539 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:16.272631 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:51:16.273035 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.274013 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.276002 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.276610 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.276723 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:16.276769 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:16.277111 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 21:51:16.277213 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:51:16.277715 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.271902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:44433 (local address 127.0.193.195:55191)
0620 21:51:16.272292 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:16.272309 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:51:16.272415 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:51:16.272730 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:51:16.272733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:51:16.272758 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:51:16.272868 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:16.272875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.274166 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.274170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:16.275807 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:51:16.275818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.276841 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.276844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:51:16.276945 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:16.276969 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:51:16.277508 (+ 539us) client_negotiation.cc:240] Negotiation successful
0620 21:51:16.277583 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":147,"threads_started":1}
I20260620 21:51:16.277712 11315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7adffe32baa4039a4ce3b9ddf4a3562" candidate_uuid: "bae85aa6fec1447ab548adeb94706616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "236084e4d4c64e4b81cdd4828c5010ef" is_pre_election: true
I20260620 21:51:16.278090 11315 raft_consensus.cc:2454] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae85aa6fec1447ab548adeb94706616 in term 0.
I20260620 21:51:16.278362 11181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7adffe32baa4039a4ce3b9ddf4a3562" candidate_uuid: "bae85aa6fec1447ab548adeb94706616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c20a5a9abd49569b90b177c6a48568" is_pre_election: true
I20260620 21:51:16.278640 11384 leader_election.cc:302] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 236084e4d4c64e4b81cdd4828c5010ef, bae85aa6fec1447ab548adeb94706616; no voters:
I20260620 21:51:16.278694 11181 raft_consensus.cc:2454] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae85aa6fec1447ab548adeb94706616 in term 0.
I20260620 21:51:16.279054 11518 raft_consensus.cc:2781] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:51:16.279173 11518 raft_consensus.cc:482] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:51:16.279258 11518 raft_consensus.cc:3037] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:51:16.280772 11518 raft_consensus.cc:504] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "236084e4d4c64e4b81cdd4828c5010ef" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34543 } } peers { permanent_uuid: "bae85aa6fec1447ab548adeb94706616" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45425 } } peers { permanent_uuid: "48c20a5a9abd49569b90b177c6a48568" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 44433 } }
I20260620 21:51:16.281219 11518 leader_election.cc:288] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [CANDIDATE]: Term 1 election: Requested vote from peers 236084e4d4c64e4b81cdd4828c5010ef (127.0.193.194:34543), 48c20a5a9abd49569b90b177c6a48568 (127.0.193.193:44433)
I20260620 21:51:16.281657 11315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7adffe32baa4039a4ce3b9ddf4a3562" candidate_uuid: "bae85aa6fec1447ab548adeb94706616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "236084e4d4c64e4b81cdd4828c5010ef"
I20260620 21:51:16.281663 11181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7adffe32baa4039a4ce3b9ddf4a3562" candidate_uuid: "bae85aa6fec1447ab548adeb94706616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c20a5a9abd49569b90b177c6a48568"
I20260620 21:51:16.281793 11315 raft_consensus.cc:3037] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:51:16.281829 11181 raft_consensus.cc:3037] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:51:16.283018 11181 raft_consensus.cc:2454] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae85aa6fec1447ab548adeb94706616 in term 1.
I20260620 21:51:16.283280 11315 raft_consensus.cc:2454] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae85aa6fec1447ab548adeb94706616 in term 1.
I20260620 21:51:16.283386 11384 leader_election.cc:302] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c20a5a9abd49569b90b177c6a48568, bae85aa6fec1447ab548adeb94706616; no voters:
I20260620 21:51:16.283577 11518 raft_consensus.cc:2781] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:51:16.283840 11518 raft_consensus.cc:686] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [term 1 LEADER]: Becoming Leader. State: Replica: bae85aa6fec1447ab548adeb94706616, State: Running, Role: LEADER
I20260620 21:51:16.284623 11518 consensus_queue.cc:237] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "236084e4d4c64e4b81cdd4828c5010ef" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34543 } } peers { permanent_uuid: "bae85aa6fec1447ab548adeb94706616" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45425 } } peers { permanent_uuid: "48c20a5a9abd49569b90b177c6a48568" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 44433 } }
I20260620 21:51:16.293150 11036 catalog_manager.cc:5495] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 reported cstate change: term changed from 0 to 1, leader changed from <none> to bae85aa6fec1447ab548adeb94706616 (127.0.193.195). New cstate: current_term: 1 leader_uuid: "bae85aa6fec1447ab548adeb94706616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "236084e4d4c64e4b81cdd4828c5010ef" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bae85aa6fec1447ab548adeb94706616" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48c20a5a9abd49569b90b177c6a48568" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 44433 } health_report { overall_health: UNKNOWN } } }
W20260620 21:51:16.316450 11496 tablet.cc:2369] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:51:16.321216 11362 tablet.cc:2369] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:51:16.327574 11510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.323223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.0.193.195:45425)
0620 21:51:16.323367 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:51:16.323371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:51:16.323386 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:51:16.323555 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:51:16.323558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:51:16.323621 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:51:16.323724 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:51:16.324077 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.324636 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.325596 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:51:16.325814 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:51:16.325905 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:51:16.326349 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:51:16.326853 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:51:16.326977 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 21:51:16.327336 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 21:51:16.327422 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:51:16.334161 11315 raft_consensus.cc:1261] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef [term 1 FOLLOWER]: Refusing update from remote peer bae85aa6fec1447ab548adeb94706616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:51:16.334560 11181 raft_consensus.cc:1261] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568 [term 1 FOLLOWER]: Refusing update from remote peer bae85aa6fec1447ab548adeb94706616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:51:16.335067 11522 consensus_queue.cc:1035] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "236084e4d4c64e4b81cdd4828c5010ef" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:51:16.335325 11518 consensus_queue.cc:1035] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48c20a5a9abd49569b90b177c6a48568" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 21:51:16.335721 11228 tablet.cc:2369] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:51:16.344105 11525 mvcc.cc:205] Tried to move back new op lower bound from 7299040363856396288 to 7299040363665649664. Current Snapshot: MvccSnapshot[applied={T|T < 7299040363856396288}]
I20260620 21:51:16.344553 11526 mvcc.cc:205] Tried to move back new op lower bound from 7299040363856396288 to 7299040363665649664. Current Snapshot: MvccSnapshot[applied={T|T < 7299040363856396288}]
I20260620 21:51:16.344703 11527 mvcc.cc:205] Tried to move back new op lower bound from 7299040363856396288 to 7299040363665649664. Current Snapshot: MvccSnapshot[applied={T|T < 7299040363856396288}]
I20260620 21:51:16.357569 11036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:51:16.357821 11036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:51:16.360567 11036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31f1c372854e487daa4568b3eb040f8f: Sending DeleteTablet for 3 replicas of tablet e7adffe32baa4039a4ce3b9ddf4a3562
I20260620 21:51:16.361555 11161 tablet_service.cc:1514] Processing DeleteTablet for tablet e7adffe32baa4039a4ce3b9ddf4a3562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:51:16 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:57014
I20260620 21:51:16.361545 11429 tablet_service.cc:1514] Processing DeleteTablet for tablet e7adffe32baa4039a4ce3b9ddf4a3562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:51:16 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:49358
I20260620 21:51:16.362112 11536 tablet_replica.cc:330] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568: stopping tablet replica
I20260620 21:51:16.362329 11295 tablet_service.cc:1514] Processing DeleteTablet for tablet e7adffe32baa4039a4ce3b9ddf4a3562 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:51:16 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:56956
I20260620 21:51:16.362614 11536 raft_consensus.cc:2229] T e7adffe32baa4039a4ce3b9ddf4a3562 P 48c20a5a9abd49569b90b177c6a48568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:51:16.362819 11538 tablet_replica.cc:330] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef: stopping tablet replica
I20260620 21:51:16.362851 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 11098
I20260620 21:51:16.363114 11538 raft_consensus.cc:2229] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:51:16.363526 11538 raft_consensus.cc:2258] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:51:16.364828 11537 tablet_replica.cc:330] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616: stopping tablet replica
I20260620 21:51:16.365249 11537 raft_consensus.cc:2229] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:51:16.365545 11538 ts_tablet_manager.cc:1905] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:51:16.365862 11537 raft_consensus.cc:2258] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:51:16.367573 11537 ts_tablet_manager.cc:1905] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:51:16.368831 11538 ts_tablet_manager.cc:1918] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:51:16.368959 11538 log.cc:1196] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/e7adffe32baa4039a4ce3b9ddf4a3562
I20260620 21:51:16.369320 11538 ts_tablet_manager.cc:1939] T e7adffe32baa4039a4ce3b9ddf4a3562 P 236084e4d4c64e4b81cdd4828c5010ef: Deleting consensus metadata
I20260620 21:51:16.370808 11022 catalog_manager.cc:4841] TS 236084e4d4c64e4b81cdd4828c5010ef (127.0.193.194:34543): tablet e7adffe32baa4039a4ce3b9ddf4a3562 (table test-table [id=4ab59bab677043df8ce2b4bee94f49db]) successfully deleted
I20260620 21:51:16.371541 11537 ts_tablet_manager.cc:1918] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:51:16.371649 11537 log.cc:1196] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/e7adffe32baa4039a4ce3b9ddf4a3562
I20260620 21:51:16.372011 11537 ts_tablet_manager.cc:1939] T e7adffe32baa4039a4ce3b9ddf4a3562 P bae85aa6fec1447ab548adeb94706616: Deleting consensus metadata
W20260620 21:51:16.372660 11022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:44433 (error 108)
I20260620 21:51:16.373215 11020 catalog_manager.cc:4841] TS bae85aa6fec1447ab548adeb94706616 (127.0.193.195:45425): tablet e7adffe32baa4039a4ce3b9ddf4a3562 (table test-table [id=4ab59bab677043df8ce2b4bee94f49db]) successfully deleted
I20260620 21:51:16.373764 11508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:51:16.373443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:44433 (local address 127.0.0.1:57016)
0620 21:51:16.373592 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 21:51:16.373665 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:44433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 21:51:16.373799 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 11232
W20260620 21:51:16.373932 11022 catalog_manager.cc:4584] TS 48c20a5a9abd49569b90b177c6a48568 (127.0.193.193:44433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7adffe32baa4039a4ce3b9ddf4a3562: Network error: Client connection negotiation failed: client connection to 127.0.193.193:44433: connect: Connection refused (error 111)
I20260620 21:51:16.379762 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 11366
I20260620 21:51:16.384877 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 11004
2026-06-20T21:51:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2537 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:51:16 dist-test-slave-kk5c krb5kdc[11542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:51:16 dist-test-slave-kk5c krb5kdc[11542](info): set up 2 sockets
Jun 20 21:51:16 dist-test-slave-kk5c krb5kdc[11542](info): commencing operation
krb5kdc: starting...
W20260620 21:51:17.512104 775 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:51:17 dist-test-slave-kk5c krb5kdc[11542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:51:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:51:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/rangeradmin_127.0.193.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/rangeradmin_127.0.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/rangerlookup_127.0.193.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/rangerlookup_127.0.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/HTTP_127.0.193.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/HTTP_127.0.193.212@KRBTEST.COM.keytab.
I20260620 21:51:17.706734 775 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBo3tyv/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/postgres -l logfile start
2026-06-20 21:51:20.538 UTC [11572] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 21:51:20.539 UTC [11572] LOG: listening on IPv4 address "127.0.193.212", port 43337
2026-06-20 21:51:20.544 UTC [11572] LOG: listening on Unix socket "/tmp/.s.PGSQL.43337"
2026-06-20 21:51:20.567 UTC [11575] LOG: database system was shut down at 2026-06-20 21:51:18 UTC
2026-06-20 21:51:20.572 UTC [11572] LOG: database system is ready to accept connections
I20260620 21:51:21.532232 775 mini_postgres.cc:96] Postgres bound to 43337
2026-06-20 21:51:21.539 UTC [11583] FATAL: database "slave" does not exist
127.0.193.212:43337 - accepting connections
I20260620 21:51:21.539981 775 mini_ranger.cc:161] Starting Ranger...
I20260620 21:51:21.557765 775 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 21:51:22.142371 775 mini_ranger.cc:87] Created ranger Postgres database
I20260620 21:51:22.142468 775 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-admin
2026-06-20 21:51:22,423 [I] DB FLAVOR :POSTGRES
2026-06-20 21:51:22,423 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 21:51:22,424 [I] Checking connection..
2026-06-20 21:51:22,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 21:51:22,691 [I] Checking connection passed.
2026-06-20 21:51:22,691 [I] --------- Verifying version history table ---------
2026-06-20 21:51:22,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:51:22,945 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 21:51:22,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:51:23,210 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 21:51:23,210 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 21:51:23,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 21:51:23.462 UTC [11647] WARNING: there is no transaction in progress
2026-06-20 21:51:23,469 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 21:51:23,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:51:23,731 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 21:51:23,731 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 21:51:23,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 21:51:24,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 21:51:24,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kk5c.c.gcp-upstream.internal','N') ;"
2026-06-20 21:51:24,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 21:51:24,756 [I] Table x_portal_user does not exist in database ranger
2026-06-20 21:51:24,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 21:51:25,019 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 21:51:25,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 21:51:25,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 21:51:25,520 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 21:51:25,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 21:51:26.620 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.639 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.659 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.678 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.697 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.728 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.797 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.805 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.822 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.837 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.852 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.872 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.885 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.894 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.908 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:26.918 UTC [11755] WARNING: there is no transaction in progress
2026-06-20 21:51:27,732 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 21:51:27,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 21:51:27,990 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 21:51:27,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 21:51:28,249 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 21:51:28,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 21:51:28,493 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 21:51:28,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 21:51:28,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 21:51:29,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kk5c.c.gcp-upstream.internal';"
2026-06-20 21:51:29,274 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 21:51:29,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 21:51:29,541 [I] DB_PATCHES have already been applied
I20260620 21:51:29.547585 775 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBo3tyv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 21:51:29.547660 775 mini_ranger.cc:193] Using host: 127.0.193.212
I20260620 21:51:29.551183 775 mini_ranger.cc:227] Ranger admin URL: http://127.0.193.212:33643
Jun 20, 2026 9:51:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:51:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:51:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:51:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:51:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:51:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:51:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 21:51:30.703804 775 mini_ranger.cc:160] Time spent starting Ranger: real 9.164s user 0.003s sys 0.008s
Jun 20, 2026 9:51:52 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 9:51:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 9:51:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 9:51:53 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 9:51:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 9:51:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 9:51:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 9:51:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 9:51:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 9:51:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 9:51:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 9:51:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 9:51:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 9:51:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 9:51:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 9:51:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 9:51:53 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 9:51:54 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 21:51:54.588951 775 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/rangerkms_127.0.193.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/rangerkms_127.0.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.193.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.193.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.193.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/HTTP_127.0.193.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.193.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/HTTP_127.0.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 21:51:54 dist-test-slave-kk5c krb5kdc[11542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992314, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 21:51:54.682303 775 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 21:51:54.693234 775 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 21:51:55.690716 775 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 21:51:55.690814 775 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-kms
I20260620 21:51:55.690915 775 mini_ranger_kms.cc:227] Using postgres at 127.0.193.212:43337
2026-06-20 21:51:55,883 [I] DB FLAVOR :POSTGRES
2026-06-20 21:51:55,884 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 21:51:55,884 [I] Checking connection
2026-06-20 21:51:55,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 21:51:56,162 [I] connection success
2026-06-20 21:51:56,163 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 21:51:56,163 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 21:51:56,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 21:51:56,446 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 21:51:56,446 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 21:51:56,446 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 21:51:56,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:43337/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 21:51:56,735 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 21:51:57.281152 775 mini_ranger_kms.cc:310] Created kms service
I20260620 21:51:57.496349 775 mini_ranger_kms.cc:326] Created kudu user
I20260620 21:51:57.540663 775 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 21:51:57 dist-test-slave-kk5c krb5kdc[11542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992314, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:51:57.938097 775 mini_ranger_kms.cc:384] Added ranger policy
I20260620 21:51:57.938418 775 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 21:51:57.938458 775 mini_ranger_kms.cc:240] Using host: 127.0.193.212
I20260620 21:51:57.942647 775 mini_ranger_kms.cc:276] Ranger KMS PID: 11936
I20260620 21:51:57.942798 775 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.193.212:57031
Jun 20, 2026 9:51:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:51:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:51:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 9:51:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 9:51:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 9:51:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 21:52:00.267634 775 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.585s user 0.015s sys 0.159s
I20260620 21:52:00.267778 775 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 21:52:00.267881 775 mini_ranger_kms.cc:401] 127.0.193.212:57031/kms/v1/keys
Jun 20 21:52:00 dist-test-slave-kk5c krb5kdc[11542](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992320, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.193.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 21:52:00 dist-test-slave-kk5c krb5kdc[11542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992320, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.193.212@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
Jun 20 21:52:01 dist-test-slave-kk5c krb5kdc[11542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992320, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.193.212@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
Jun 20 21:52:03 dist-test-slave-kk5c krb5kdc[11542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:52:03.699774 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:46853
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:46853
--ranger_config_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.193.212:57031/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:52:03.830611 11986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:52:03.830940 11986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:52:03.830981 11986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:52:03.835480 11986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:52:03.835582 11986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:52:03.835603 11986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:52:03.835620 11986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:52:03.835636 11986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:52:03.843953 11986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.193.212:57031/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:46853
--ranger_config_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:46853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:52:03.845127 11986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:52:03.846177 11986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:52:03.855993 11994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:03.856011 11991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:03.856011 11992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:03.856099 11986 server_base.cc:1034] running on GCE node
I20260620 21:52:03.856935 11986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:52:03.858177 11986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:52:03.859383 11986 hybrid_clock.cc:648] HybridClock initialized: now 1781992323859354 us; error 64 us; skew 500 ppm
Jun 20 21:52:03 dist-test-slave-kk5c krb5kdc[11542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:52:03.865975 11986 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:52:03.867473 11986 webserver.cc:460] Webserver started at http://127.0.193.254:35277/ using document root <none> and password file <none>
I20260620 21:52:03.868160 11986 fs_manager.cc:359] Metadata directory not provided
I20260620 21:52:03.868258 11986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:52:03.868507 11986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:52:03 dist-test-slave-kk5c krb5kdc[11542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:03.931565 11986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "f4ff7fb0d1c74cab889f8c2135b13d46"
format_stamp: "Formatted at 2026-06-20 21:52:03 on dist-test-slave-kk5c"
server_key: "68651ff5d589f867b968e49aa9d0eb58"
server_key_iv: "912c3816925323a39b745ba89df1f2e7"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:03.932760 11986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "f4ff7fb0d1c74cab889f8c2135b13d46"
format_stamp: "Formatted at 2026-06-20 21:52:03 on dist-test-slave-kk5c"
server_key: "68651ff5d589f867b968e49aa9d0eb58"
server_key_iv: "912c3816925323a39b745ba89df1f2e7"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:03.938632 11986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 21:52:03.958117 12001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:03.959987 11986 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 21:52:03.960166 11986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "f4ff7fb0d1c74cab889f8c2135b13d46"
format_stamp: "Formatted at 2026-06-20 21:52:03 on dist-test-slave-kk5c"
server_key: "68651ff5d589f867b968e49aa9d0eb58"
server_key_iv: "912c3816925323a39b745ba89df1f2e7"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:03.960325 11986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:52:03.985877 11986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:52:03.989078 11986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:52:03.989324 11986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:52:03.997648 11986 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:46853
I20260620 21:52:03.997670 12053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:46853 every 8 connection(s)
I20260620 21:52:03.998684 11986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:52:03.999907 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 11986
I20260620 21:52:04.000051 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:52:04.000406 775 external_mini_cluster.cc:1428] Setting key 424f35dfffa3d24d9342ceb083fac172
I20260620 21:52:04.002614 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:52:04.009968 12054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46: Bootstrap starting.
Jun 20 21:52:04 dist-test-slave-kk5c krb5kdc[11542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:52:04.013947 12054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46: Neither blocks nor log segments found. Creating new log.
I20260620 21:52:04.015368 12054 log.cc:824] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46: Log is configured to *not* fsync() on all Append() calls
I20260620 21:52:04.017148 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:04.001986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46288 (local address 127.0.193.254:46853)
0620 21:52:04.002471 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:04.002501 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:04.002542 (+ 41us) server_negotiation.cc:407] Connection header received
0620 21:52:04.003359 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:04.003378 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:04.003747 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:04.004059 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:52:04.004973 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:04.006060 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:04.007311 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:04.007972 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:04.008155 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:04.011838 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:52:04.011845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:52:04.011863 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:52:04.011900 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:52:04.013862 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:04.014511 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:04.014518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:04.014526 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:04.014678 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:04.014998 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:04.015001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:04.015002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:04.015577 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:52:04.015794 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:04.016076 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 21:52:04.016374 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":200,"threads_started":1}
I20260620 21:52:04.017715 12054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46: No bootstrap required, opened a new log
I20260620 21:52:04.021318 12054 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ff7fb0d1c74cab889f8c2135b13d46" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46853 } }
I20260620 21:52:04.021606 12054 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:52:04.021697 12054 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4ff7fb0d1c74cab889f8c2135b13d46, State: Initialized, Role: FOLLOWER
I20260620 21:52:04.022199 12054 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ff7fb0d1c74cab889f8c2135b13d46" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46853 } }
I20260620 21:52:04.022365 12054 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:52:04.022457 12054 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:52:04.022564 12054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:52:04.023694 12054 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ff7fb0d1c74cab889f8c2135b13d46" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46853 } }
I20260620 21:52:04.024070 12054 leader_election.cc:302] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4ff7fb0d1c74cab889f8c2135b13d46; no voters:
I20260620 21:52:04.024511 12054 leader_election.cc:288] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:52:04.024662 12059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:52:04.024947 12059 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [term 1 LEADER]: Becoming Leader. State: Replica: f4ff7fb0d1c74cab889f8c2135b13d46, State: Running, Role: LEADER
I20260620 21:52:04.025264 12059 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ff7fb0d1c74cab889f8c2135b13d46" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46853 } }
I20260620 21:52:04.025856 12054 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:52:04.027055 12061 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4ff7fb0d1c74cab889f8c2135b13d46. Latest consensus state: current_term: 1 leader_uuid: "f4ff7fb0d1c74cab889f8c2135b13d46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ff7fb0d1c74cab889f8c2135b13d46" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46853 } } }
I20260620 21:52:04.027087 12060 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4ff7fb0d1c74cab889f8c2135b13d46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ff7fb0d1c74cab889f8c2135b13d46" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 46853 } } }
I20260620 21:52:04.027210 12061 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [sys.catalog]: This master's current role is: LEADER
I20260620 21:52:04.027222 12060 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46 [sys.catalog]: This master's current role is: LEADER
I20260620 21:52:04.027653 12068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:52:04.030925 12068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:52:04.031529 12054 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 21:52:04.042346 12068 catalog_manager.cc:1347] Generated new cluster ID: 713a47a3e14b474dab98263a97b0f5b8
I20260620 21:52:04.042438 12068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:52:04.058665 12068 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:52:04.059857 12068 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:52:04.072124 12068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46: Generated new TSK 0
I20260620 21:52:04.072923 12068 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:52:04.824847 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.193.212:57031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46853
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:52:04.989378 12099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:52:04.989879 12099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:52:04.989998 12099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:52:04.995599 12099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:52:04.995759 12099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:52:04.995913 12099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:52:05.002454 12099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.193.212:57031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:52:05.003695 12099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:52:05.004767 12099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:52:05.019881 12106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:05.019886 12099 server_base.cc:1034] running on GCE node
W20260620 21:52:05.019904 12109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:05.020316 12107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:05.020925 12099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:52:05.021688 12099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:52:05.022876 12099 hybrid_clock.cc:648] HybridClock initialized: now 1781992325022847 us; error 56 us; skew 500 ppm
Jun 20 21:52:05 dist-test-slave-kk5c krb5kdc[11542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:52:05.026686 12099 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:52:05.028556 12099 webserver.cc:460] Webserver started at http://127.0.193.193:36807/ using document root <none> and password file <none>
I20260620 21:52:05.029897 12099 fs_manager.cc:359] Metadata directory not provided
I20260620 21:52:05.030031 12099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:52:05.030351 12099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:52:05 dist-test-slave-kk5c krb5kdc[11542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:05.051661 12099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "1965f61888f549f28764af765c122202"
format_stamp: "Formatted at 2026-06-20 21:52:05 on dist-test-slave-kk5c"
server_key: "b181d162d516355d98dcedf31e27c0a1"
server_key_iv: "097396e8df6c389277b628275a02a57e"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:05.052474 12099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "1965f61888f549f28764af765c122202"
format_stamp: "Formatted at 2026-06-20 21:52:05 on dist-test-slave-kk5c"
server_key: "b181d162d516355d98dcedf31e27c0a1"
server_key_iv: "097396e8df6c389277b628275a02a57e"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:05.059094 12099 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 21:52:05.091720 12116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:05.093638 12099 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260620 21:52:05.093870 12099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "1965f61888f549f28764af765c122202"
format_stamp: "Formatted at 2026-06-20 21:52:05 on dist-test-slave-kk5c"
server_key: "b181d162d516355d98dcedf31e27c0a1"
server_key_iv: "097396e8df6c389277b628275a02a57e"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:05.094132 12099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:52:05.114614 12099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:52:05.118502 12099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:52:05.118768 12099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:52:05.119599 12099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:52:05.120817 12099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:52:05.120884 12099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:05.120947 12099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:52:05.120975 12099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:05.134271 12099 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:41981
I20260620 21:52:05.134383 12231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:41981 every 8 connection(s)
I20260620 21:52:05.136013 12099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:52:05.143399 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 12099
I20260620 21:52:05.143541 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:52:05.143930 775 external_mini_cluster.cc:1428] Setting key 9babfb48ff3c1f77b2f6c7d9340dea8b
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
Jun 20 21:52:05 dist-test-slave-kk5c krb5kdc[11542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:52:05.170228 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:05.150392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:46853 (local address 127.0.193.193:53141)
0620 21:52:05.152698 (+ 2306us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:05.152762 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:05.153861 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:05.154868 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:05.154879 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:05.155275 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:52:05.155969 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:05.155987 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.157157 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.157161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:05.158585 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:05.158598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.159436 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.159440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:05.159642 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:05.160443 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:52:05.160466 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:52:05.163012 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:52:05.165864 (+ 2852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:05.165872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:05.165893 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:05.166309 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:05.167087 (+ 778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:05.167090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:05.167092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:05.167356 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:05.167984 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:52:05.167993 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:52:05.168381 (+ 388us) client_negotiation.cc:769] Sending connection context
0620 21:52:05.168611 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 21:52:05.169035 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2056,"thread_start_us":195,"threads_started":1}
I20260620 21:52:05.171339 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:05.151422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:53141 (local address 127.0.193.254:46853)
0620 21:52:05.154317 (+ 2895us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:05.154322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:05.154338 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:52:05.154385 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:05.154388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:05.154436 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:05.154528 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:52:05.156145 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.156993 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.158740 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.159103 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.159239 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:05.163323 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:52:05.163328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:52:05.163331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:52:05.163369 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:52:05.165668 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:05.166449 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:05.166453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:05.166455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:05.166954 (+ 499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:05.167515 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:05.167520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:05.167522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:05.167807 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:52:05.167919 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:05.170893 (+ 2974us) server_negotiation.cc:299] Negotiation successful
0620 21:52:05.171056 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2810,"thread_start_us":64,"threads_started":1}
I20260620 21:52:05.172850 12232 heartbeater.cc:347] Connected to a master server at 127.0.193.254:46853
I20260620 21:52:05.173130 12232 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:05.173789 12232 heartbeater.cc:511] Master 127.0.193.254:46853 requested a full tablet report, sending...
I20260620 21:52:05.175730 12018 ts_manager.cc:194] Registered new tserver with Master: 1965f61888f549f28764af765c122202 (127.0.193.193:41981)
I20260620 21:52:05.177466 12018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:53141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:52:05.188287 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.193.212:57031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46853
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:52:05 dist-test-slave-kk5c krb5kdc[11542](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 21:52:05.341406 12238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:52:05.341777 12238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:52:05.341881 12238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:52:05.346470 12238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:52:05.346570 12238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:52:05.346685 12238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:52:05.353122 12238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.193.212:57031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:52:05.354362 12238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:52:05.355540 12238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:52:05.366822 12245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:05.366822 12244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:05.368084 12238 server_base.cc:1034] running on GCE node
W20260620 21:52:05.366855 12247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:05.368683 12238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:52:05.369436 12238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:52:05.370622 12238 hybrid_clock.cc:648] HybridClock initialized: now 1781992325370583 us; error 54 us; skew 500 ppm
Jun 20 21:52:05 dist-test-slave-kk5c krb5kdc[11542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:52:05.374277 12238 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:52:05.375522 12238 webserver.cc:460] Webserver started at http://127.0.193.194:36315/ using document root <none> and password file <none>
I20260620 21:52:05.376176 12238 fs_manager.cc:359] Metadata directory not provided
I20260620 21:52:05.376281 12238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:52:05.376565 12238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:52:05 dist-test-slave-kk5c krb5kdc[11542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:05.398130 12238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "8c98a06590244652af8b98add9c6e004"
format_stamp: "Formatted at 2026-06-20 21:52:05 on dist-test-slave-kk5c"
server_key: "6e8f32b4676791f2c6bb35f7e76e72e6"
server_key_iv: "b79484e0c8cdd27387a4ce01d86aa204"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:05.398928 12238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "8c98a06590244652af8b98add9c6e004"
format_stamp: "Formatted at 2026-06-20 21:52:05 on dist-test-slave-kk5c"
server_key: "6e8f32b4676791f2c6bb35f7e76e72e6"
server_key_iv: "b79484e0c8cdd27387a4ce01d86aa204"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:05.403913 12238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 21:52:05.427421 12254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:05.429152 12238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:52:05.429339 12238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "8c98a06590244652af8b98add9c6e004"
format_stamp: "Formatted at 2026-06-20 21:52:05 on dist-test-slave-kk5c"
server_key: "6e8f32b4676791f2c6bb35f7e76e72e6"
server_key_iv: "b79484e0c8cdd27387a4ce01d86aa204"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:05.429533 12238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:52:05.446856 12238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:52:05.451061 12238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:52:05.451334 12238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:52:05.452064 12238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:52:05.453145 12238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:52:05.453207 12238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:05.453261 12238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:52:05.453293 12238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:05.470145 12238 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:35879
I20260620 21:52:05.472069 12367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:35879 every 8 connection(s)
I20260620 21:52:05.472807 12238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:52:05.478260 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 12238
I20260620 21:52:05.478403 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:52:05.478701 775 external_mini_cluster.cc:1428] Setting key 44a5189e4d4dbbd8ec911fddcd4458cc
Jun 20 21:52:05 dist-test-slave-kk5c krb5kdc[11542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:52:05.499640 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:05.475450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:44211 (local address 127.0.193.254:46853)
0620 21:52:05.475606 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:05.475612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:05.483068 (+ 7456us) server_negotiation.cc:407] Connection header received
0620 21:52:05.485785 (+ 2717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:05.485790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:05.485856 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:05.485917 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:52:05.487933 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.488798 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.490685 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.491071 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.491168 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:05.494922 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:52:05.494927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:52:05.494930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:52:05.494962 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:52:05.496853 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:05.497592 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:05.497599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:05.497602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:05.497725 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:05.498188 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:05.498195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:05.498197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:05.498471 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:52:05.498563 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:05.499297 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 21:52:05.499441 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
I20260620 21:52:05.507123 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:05.475962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:46853 (local address 127.0.193.194:44211)
0620 21:52:05.482836 (+ 6874us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:05.482934 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:05.484261 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:05.486447 (+ 2186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:05.486459 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:05.486898 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:52:05.487706 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:05.487730 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.488989 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.488992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:05.490519 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:05.490532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.491435 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.491440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:05.491662 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:05.492866 (+ 1204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:52:05.492891 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:52:05.494662 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:52:05.497002 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:05.497009 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:05.497029 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:05.497452 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:05.497857 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:05.497861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:05.497865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:05.498075 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:05.498638 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:52:05.498647 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:52:05.498946 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 21:52:05.499196 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 21:52:05.499566 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1863,"thread_start_us":150,"threads_started":1}
I20260620 21:52:05.509025 12368 heartbeater.cc:347] Connected to a master server at 127.0.193.254:46853
I20260620 21:52:05.509384 12368 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
I20260620 21:52:05.510072 12368 heartbeater.cc:511] Master 127.0.193.254:46853 requested a full tablet report, sending...
I20260620 21:52:05.512338 12018 ts_manager.cc:194] Registered new tserver with Master: 8c98a06590244652af8b98add9c6e004 (127.0.193.194:35879)
I20260620 21:52:05.512955 12018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:44211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:52:05.528292 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.193.212:57031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46853
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:52:05.711048 12373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:52:05.711601 12373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:52:05.711730 12373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Jun 20 21:52:05 dist-test-slave-kk5c krb5kdc[11542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
W20260620 21:52:05.719377 12373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:52:05.719583 12373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:52:05.719730 12373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:52:05.728008 12373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.193.212:57031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:46853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:52:05.729197 12373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:52:05.730558 12373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:52:05.759717 12381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:05.760260 12378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:05.760802 12379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:05.762553 12373 server_base.cc:1034] running on GCE node
I20260620 21:52:05.763329 12373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:52:05.764130 12373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:52:05.779719 12373 hybrid_clock.cc:648] HybridClock initialized: now 1781992325779603 us; error 109 us; skew 500 ppm
Jun 20 21:52:05 dist-test-slave-kk5c krb5kdc[11542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:52:05.783967 12373 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:52:05.785233 12373 webserver.cc:460] Webserver started at http://127.0.193.195:45911/ using document root <none> and password file <none>
I20260620 21:52:05.786003 12373 fs_manager.cc:359] Metadata directory not provided
I20260620 21:52:05.786110 12373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:52:05.786399 12373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:52:05 dist-test-slave-kk5c krb5kdc[11542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:05.804060 12373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "cf01d03e0feb446faa6648a235eda07d"
format_stamp: "Formatted at 2026-06-20 21:52:05 on dist-test-slave-kk5c"
server_key: "43803afd73a32156c00eae5b6e00e274"
server_key_iv: "f7034b11bd160266af6184b9972e605d"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:05.804929 12373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "cf01d03e0feb446faa6648a235eda07d"
format_stamp: "Formatted at 2026-06-20 21:52:05 on dist-test-slave-kk5c"
server_key: "43803afd73a32156c00eae5b6e00e274"
server_key_iv: "f7034b11bd160266af6184b9972e605d"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:05.812798 12373 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260620 21:52:05.847669 12388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:05.849663 12373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 21:52:05.849830 12373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "cf01d03e0feb446faa6648a235eda07d"
format_stamp: "Formatted at 2026-06-20 21:52:05 on dist-test-slave-kk5c"
server_key: "43803afd73a32156c00eae5b6e00e274"
server_key_iv: "f7034b11bd160266af6184b9972e605d"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:05.849982 12373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:52:05.865171 12373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:52:05.868829 12373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:52:05.869036 12373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:52:05.869752 12373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:52:05.870790 12373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:52:05.870865 12373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:05.870936 12373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:52:05.870968 12373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:05.882990 12373 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:37235
I20260620 21:52:05.883029 12501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:37235 every 8 connection(s)
I20260620 21:52:05.884770 12373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:52:05.889916 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 12373
I20260620 21:52:05.890058 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:52:05.890336 775 external_mini_cluster.cc:1428] Setting key 69aa10d759890b7cea248471442ac85e
Jun 20 21:52:05 dist-test-slave-kk5c krb5kdc[11542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:52:05.903996 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:05.887025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:55141 (local address 127.0.193.254:46853)
0620 21:52:05.887164 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:05.887169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:05.888508 (+ 1339us) server_negotiation.cc:407] Connection header received
0620 21:52:05.889891 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:05.889895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:05.889956 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:05.890011 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:52:05.892107 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.892933 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.894771 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.895155 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.895319 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:05.898395 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:52:05.898400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:52:05.898403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:52:05.898434 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:52:05.901080 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:05.901914 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:05.901918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:05.901920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:05.902014 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:05.902461 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:05.902464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:05.902466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:05.902794 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:52:05.902865 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:05.903660 (+ 795us) server_negotiation.cc:299] Negotiation successful
0620 21:52:05.903830 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:52:05.905606 12504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:05.887613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:46853 (local address 127.0.193.195:55141)
0620 21:52:05.888232 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:05.888303 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:05.889583 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:05.890466 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:05.890476 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:05.890856 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:52:05.891917 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:05.891941 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.893110 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.893114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:05.894581 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:05.894596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.895373 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.895377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:05.895594 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:05.896387 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:52:05.896417 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:52:05.898150 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:52:05.901254 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:05.901262 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:05.901348 (+ 86us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:05.901764 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:05.902144 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:05.902147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:05.902149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:05.902347 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:05.902948 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:52:05.902956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:52:05.903365 (+ 409us) client_negotiation.cc:769] Sending connection context
0620 21:52:05.904158 (+ 793us) client_negotiation.cc:240] Negotiation successful
0620 21:52:05.904542 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":183,"threads_started":1}
I20260620 21:52:05.907310 12502 heartbeater.cc:347] Connected to a master server at 127.0.193.254:46853
I20260620 21:52:05.907574 12502 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:05.908202 12502 heartbeater.cc:511] Master 127.0.193.254:46853 requested a full tablet report, sending...
I20260620 21:52:05.909579 12018 ts_manager.cc:194] Registered new tserver with Master: cf01d03e0feb446faa6648a235eda07d (127.0.193.195:37235)
I20260620 21:52:05.910403 12018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:55141
I20260620 21:52:05.917289 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:52:05.930308 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:05.919738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52090 (local address 127.0.193.254:46853)
0620 21:52:05.919860 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:05.919864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:05.920147 (+ 283us) server_negotiation.cc:407] Connection header received
0620 21:52:05.920272 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:05.920274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:05.920328 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:05.920413 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:52:05.921352 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.922155 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.923547 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.923901 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.923990 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:05.926136 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:52:05.926141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:52:05.926144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:52:05.926179 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:52:05.928249 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:05.928800 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:05.928804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:05.928806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:05.928905 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:05.929229 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:05.929233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:05.929235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:05.929490 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:52:05.929630 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:05.929940 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 21:52:05.930077 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 21:52:05.934629 12018 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:52:05.939845 12018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:52:05.958818 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:05.952202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:41981 (local address 127.0.0.1:41166)
0620 21:52:05.952959 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:05.952972 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:05.953041 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:05.953699 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:05.953702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:05.953719 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:52:05.953816 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:05.953821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.955452 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.955456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:05.956846 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:05.956870 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.958160 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.958164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:05.958301 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:05.958356 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 21:52:05.958457 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 21:52:05.958596 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":94,"threads_started":1}
I20260620 21:52:05.960203 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:05.952413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.0.193.193:41981)
0620 21:52:05.953271 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:05.953277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:05.953298 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:52:05.953371 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:05.953375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:05.953552 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:05.953976 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:52:05.954104 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.955317 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.957026 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.957976 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.959682 (+ 1706us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:05.959888 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:05.959949 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 21:52:05.960051 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":84,"threads_started":1}
I20260620 21:52:05.960813 12514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:05.950982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:35879 (local address 127.0.0.1:59346)
0620 21:52:05.952479 (+ 1497us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:05.952543 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:05.952718 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:05.953369 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:05.953375 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:05.953399 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:52:05.953546 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:05.953552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.956872 (+ 3320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.956876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:05.958103 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:05.958116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.960377 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.960381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:05.960476 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:05.960502 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:52:05.960558 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 21:52:05.960647 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"thread_start_us":126,"threads_started":1}
I20260620 21:52:05.961449 12515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:05.951058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59346 (local address 127.0.193.194:35879)
0620 21:52:05.951534 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:05.951549 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:05.952642 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 21:52:05.952912 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:05.952920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:05.953162 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:05.954536 (+ 1374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:52:05.954680 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.956701 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.959318 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.960220 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.960999 (+ 779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:05.961147 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:05.961195 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 21:52:05.961288 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":125,"threads_started":1}
I20260620 21:52:05.963740 12155 tablet_service.cc:1467] Processing CreateTablet for tablet 046587af41ab46dbae6e6562e88927a3 (DEFAULT_TABLE table=test-table [id=f19dc475bc4f4cfd82d0e0d25b635814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:52:05.965098 12155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046587af41ab46dbae6e6562e88927a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:52:05.965539 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:05.952519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:37235 (local address 127.0.0.1:58230)
0620 21:52:05.959467 (+ 6948us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:05.959481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:05.959552 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:05.961490 (+ 1938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:05.961493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:05.961514 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:52:05.961609 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:05.961617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.962858 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.962861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:05.963869 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:05.963879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.965104 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.965107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:05.965194 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:05.965218 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:52:05.965315 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 21:52:05.965410 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6887,"thread_start_us":98,"threads_started":1}
I20260620 21:52:05.967912 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:05.952838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.0.193.195:37235)
0620 21:52:05.960928 (+ 8090us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:05.960943 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:05.960978 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:52:05.961062 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:05.961068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:05.961315 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:05.961759 (+ 444us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:52:05.961871 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.962734 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.964016 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:05.964909 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:05.965102 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:05.965248 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:05.967602 (+ 2354us) server_negotiation.cc:299] Negotiation successful
0620 21:52:05.967735 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7965,"thread_start_us":113,"threads_started":1}
I20260620 21:52:05.973523 12302 tablet_service.cc:1467] Processing CreateTablet for tablet 046587af41ab46dbae6e6562e88927a3 (DEFAULT_TABLE table=test-table [id=f19dc475bc4f4cfd82d0e0d25b635814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:52:05.974797 12302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046587af41ab46dbae6e6562e88927a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:52:05.975402 12422 tablet_service.cc:1467] Processing CreateTablet for tablet 046587af41ab46dbae6e6562e88927a3 (DEFAULT_TABLE table=test-table [id=f19dc475bc4f4cfd82d0e0d25b635814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:52:05.976586 12422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046587af41ab46dbae6e6562e88927a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:52:05.987433 12522 tablet_bootstrap.cc:492] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202: Bootstrap starting.
I20260620 21:52:05.988358 12521 tablet_bootstrap.cc:492] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d: Bootstrap starting.
I20260620 21:52:05.990002 12522 tablet_bootstrap.cc:654] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202: Neither blocks nor log segments found. Creating new log.
I20260620 21:52:05.990936 12522 log.cc:824] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202: Log is configured to *not* fsync() on all Append() calls
I20260620 21:52:05.991590 12520 tablet_bootstrap.cc:492] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004: Bootstrap starting.
I20260620 21:52:05.993531 12522 tablet_bootstrap.cc:492] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202: No bootstrap required, opened a new log
I20260620 21:52:05.993772 12522 ts_tablet_manager.cc:1397] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 21:52:05.995065 12520 tablet_bootstrap.cc:654] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004: Neither blocks nor log segments found. Creating new log.
I20260620 21:52:05.995996 12520 log.cc:824] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004: Log is configured to *not* fsync() on all Append() calls
I20260620 21:52:05.996554 12521 tablet_bootstrap.cc:654] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d: Neither blocks nor log segments found. Creating new log.
I20260620 21:52:05.997625 12521 log.cc:824] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:52:05.998513 12522 raft_consensus.cc:348] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c98a06590244652af8b98add9c6e004" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35879 } } peers { permanent_uuid: "1965f61888f549f28764af765c122202" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41981 } } peers { permanent_uuid: "cf01d03e0feb446faa6648a235eda07d" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37235 } }
I20260620 21:52:05.998807 12522 raft_consensus.cc:374] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:52:05.998859 12522 raft_consensus.cc:729] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1965f61888f549f28764af765c122202, State: Initialized, Role: FOLLOWER
I20260620 21:52:05.999603 12522 consensus_queue.cc:260] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c98a06590244652af8b98add9c6e004" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35879 } } peers { permanent_uuid: "1965f61888f549f28764af765c122202" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41981 } } peers { permanent_uuid: "cf01d03e0feb446faa6648a235eda07d" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37235 } }
I20260620 21:52:06.000833 12522 ts_tablet_manager.cc:1428] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260620 21:52:06.001353 12232 heartbeater.cc:503] Master 127.0.193.254:46853 was elected leader, sending a full tablet report...
I20260620 21:52:06.009114 12520 tablet_bootstrap.cc:492] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004: No bootstrap required, opened a new log
I20260620 21:52:06.009361 12520 ts_tablet_manager.cc:1397] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260620 21:52:06.012522 12521 tablet_bootstrap.cc:492] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d: No bootstrap required, opened a new log
I20260620 21:52:06.012764 12521 ts_tablet_manager.cc:1397] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.005s
I20260620 21:52:06.013484 12520 raft_consensus.cc:348] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c98a06590244652af8b98add9c6e004" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35879 } } peers { permanent_uuid: "1965f61888f549f28764af765c122202" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41981 } } peers { permanent_uuid: "cf01d03e0feb446faa6648a235eda07d" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37235 } }
I20260620 21:52:06.013783 12520 raft_consensus.cc:374] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:52:06.013846 12520 raft_consensus.cc:729] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c98a06590244652af8b98add9c6e004, State: Initialized, Role: FOLLOWER
I20260620 21:52:06.014335 12520 consensus_queue.cc:260] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c98a06590244652af8b98add9c6e004" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35879 } } peers { permanent_uuid: "1965f61888f549f28764af765c122202" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41981 } } peers { permanent_uuid: "cf01d03e0feb446faa6648a235eda07d" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37235 } }
I20260620 21:52:06.016068 12520 ts_tablet_manager.cc:1428] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 21:52:06.016494 12368 heartbeater.cc:503] Master 127.0.193.254:46853 was elected leader, sending a full tablet report...
I20260620 21:52:06.019593 12521 raft_consensus.cc:348] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c98a06590244652af8b98add9c6e004" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35879 } } peers { permanent_uuid: "1965f61888f549f28764af765c122202" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41981 } } peers { permanent_uuid: "cf01d03e0feb446faa6648a235eda07d" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37235 } }
I20260620 21:52:06.019960 12521 raft_consensus.cc:374] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:52:06.020020 12521 raft_consensus.cc:729] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf01d03e0feb446faa6648a235eda07d, State: Initialized, Role: FOLLOWER
I20260620 21:52:06.020539 12521 consensus_queue.cc:260] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c98a06590244652af8b98add9c6e004" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35879 } } peers { permanent_uuid: "1965f61888f549f28764af765c122202" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41981 } } peers { permanent_uuid: "cf01d03e0feb446faa6648a235eda07d" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37235 } }
I20260620 21:52:06.021713 12521 ts_tablet_manager.cc:1428] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260620 21:52:06.022096 12502 heartbeater.cc:503] Master 127.0.193.254:46853 was elected leader, sending a full tablet report...
I20260620 21:52:06.059487 12527 raft_consensus.cc:482] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:52:06.059754 12527 raft_consensus.cc:504] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c98a06590244652af8b98add9c6e004" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35879 } } peers { permanent_uuid: "1965f61888f549f28764af765c122202" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41981 } } peers { permanent_uuid: "cf01d03e0feb446faa6648a235eda07d" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37235 } }
I20260620 21:52:06.061267 12527 leader_election.cc:288] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1965f61888f549f28764af765c122202 (127.0.193.193:41981), cf01d03e0feb446faa6648a235eda07d (127.0.193.195:37235)
I20260620 21:52:06.061342 12526 raft_consensus.cc:482] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:52:06.061553 12526 raft_consensus.cc:504] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c98a06590244652af8b98add9c6e004" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35879 } } peers { permanent_uuid: "1965f61888f549f28764af765c122202" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41981 } } peers { permanent_uuid: "cf01d03e0feb446faa6648a235eda07d" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37235 } }
I20260620 21:52:06.062856 12526 leader_election.cc:288] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c98a06590244652af8b98add9c6e004 (127.0.193.194:35879), cf01d03e0feb446faa6648a235eda07d (127.0.193.195:37235)
Jun 20 21:52:06 dist-test-slave-kk5c krb5kdc[11542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:06.075238 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:06.063585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:37235 (local address 127.0.193.193:46017)
0620 21:52:06.070763 (+ 7178us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:06.070779 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:06.070903 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:06.071326 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:06.071329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:06.071346 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:52:06.071481 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:06.071487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.072518 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.072522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:06.073677 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:06.073688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.074755 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.074759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:06.074856 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:06.074879 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:52:06.074951 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 21:52:06.075024 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7096,"thread_start_us":97,"threads_started":1}
I20260620 21:52:06.075670 12534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:06.064290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:46017 (local address 127.0.193.195:37235)
0620 21:52:06.064872 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:06.064881 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:06.070998 (+ 6117us) server_negotiation.cc:407] Connection header received
0620 21:52:06.071060 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:06.071063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:06.071111 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:06.071210 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:52:06.071607 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.072375 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.073825 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.074526 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.074629 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:06.074683 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:06.075434 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 21:52:06.075512 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":113,"threads_started":1}
I20260620 21:52:06.076768 12448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046587af41ab46dbae6e6562e88927a3" candidate_uuid: "1965f61888f549f28764af765c122202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf01d03e0feb446faa6648a235eda07d" is_pre_election: true
I20260620 21:52:06.077112 12448 raft_consensus.cc:2454] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1965f61888f549f28764af765c122202 in term 0.
I20260620 21:52:06.077739 12117 leader_election.cc:302] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1965f61888f549f28764af765c122202, cf01d03e0feb446faa6648a235eda07d; no voters:
I20260620 21:52:06.078099 12526 raft_consensus.cc:2781] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:52:06.078183 12526 raft_consensus.cc:482] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:52:06.078233 12526 raft_consensus.cc:3037] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:52:06.079803 12526 raft_consensus.cc:504] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c98a06590244652af8b98add9c6e004" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35879 } } peers { permanent_uuid: "1965f61888f549f28764af765c122202" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41981 } } peers { permanent_uuid: "cf01d03e0feb446faa6648a235eda07d" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37235 } }
I20260620 21:52:06.080296 12526 leader_election.cc:288] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [CANDIDATE]: Term 1 election: Requested vote from peers 8c98a06590244652af8b98add9c6e004 (127.0.193.194:35879), cf01d03e0feb446faa6648a235eda07d (127.0.193.195:37235)
I20260620 21:52:06.080888 12448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046587af41ab46dbae6e6562e88927a3" candidate_uuid: "1965f61888f549f28764af765c122202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf01d03e0feb446faa6648a235eda07d"
I20260620 21:52:06.081049 12448 raft_consensus.cc:3037] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:52:06.082509 12448 raft_consensus.cc:2454] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1965f61888f549f28764af765c122202 in term 1.
I20260620 21:52:06.082911 12117 leader_election.cc:302] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1965f61888f549f28764af765c122202, cf01d03e0feb446faa6648a235eda07d; no voters:
I20260620 21:52:06.083123 12526 raft_consensus.cc:2781] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:52:06.083412 12526 raft_consensus.cc:686] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [term 1 LEADER]: Becoming Leader. State: Replica: 1965f61888f549f28764af765c122202, State: Running, Role: LEADER
I20260620 21:52:06.083588 12515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:06.064596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:50293 (local address 127.0.193.194:35879)
0620 21:52:06.064751 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:06.064756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:06.065136 (+ 380us) server_negotiation.cc:407] Connection header received
0620 21:52:06.065914 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:06.065917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:06.065977 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:06.066027 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:52:06.067791 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.068692 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.082183 (+ 13491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.082784 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.082881 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:06.082931 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:06.083369 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 21:52:06.083463 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 21:52:06.083864 12526 consensus_queue.cc:237] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c98a06590244652af8b98add9c6e004" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35879 } } peers { permanent_uuid: "1965f61888f549f28764af765c122202" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41981 } } peers { permanent_uuid: "cf01d03e0feb446faa6648a235eda07d" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37235 } }
I20260620 21:52:06.084021 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:06.063240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:35879 (local address 127.0.193.193:50293)
0620 21:52:06.065037 (+ 1797us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:06.065053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:06.065573 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:06.066452 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:06.066455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:06.066474 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:52:06.066609 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:06.066616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.080827 (+ 14211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.080830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:06.082039 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:06.082049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.083035 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.083038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:06.083138 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:06.083161 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:52:06.083735 (+ 574us) client_negotiation.cc:240] Negotiation successful
0620 21:52:06.083819 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1728,"thread_start_us":96,"threads_started":1}
I20260620 21:52:06.084712 12321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046587af41ab46dbae6e6562e88927a3" candidate_uuid: "1965f61888f549f28764af765c122202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c98a06590244652af8b98add9c6e004" is_pre_election: true
I20260620 21:52:06.084780 12322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046587af41ab46dbae6e6562e88927a3" candidate_uuid: "1965f61888f549f28764af765c122202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c98a06590244652af8b98add9c6e004"
I20260620 21:52:06.084928 12321 raft_consensus.cc:2454] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1965f61888f549f28764af765c122202 in term 0.
I20260620 21:52:06.086005 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:06.063900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:45925 (local address 127.0.193.195:37235)
0620 21:52:06.064163 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:06.064168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:06.073867 (+ 9699us) server_negotiation.cc:407] Connection header received
0620 21:52:06.074057 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:06.074060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:06.074102 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:06.074144 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:52:06.074531 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.075242 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.084688 (+ 9446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.085307 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.085431 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:06.085477 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:06.085823 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 21:52:06.085920 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260620 21:52:06.086447 12531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:06.062171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:37235 (local address 127.0.193.194:45925)
0620 21:52:06.073775 (+ 11604us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:06.073793 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:06.073949 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:06.074260 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:06.074263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:06.074279 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:52:06.074404 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:06.074411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.075383 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.075386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:06.076914 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:06.076926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.085574 (+ 8648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.085577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:06.085680 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:06.085703 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:52:06.086214 (+ 511us) client_negotiation.cc:240] Negotiation successful
0620 21:52:06.086298 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11502,"thread_start_us":103,"threads_started":1}
I20260620 21:52:06.086977 12448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046587af41ab46dbae6e6562e88927a3" candidate_uuid: "8c98a06590244652af8b98add9c6e004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf01d03e0feb446faa6648a235eda07d" is_pre_election: true
I20260620 21:52:06.087862 12014 catalog_manager.cc:5495] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1965f61888f549f28764af765c122202 (127.0.193.193). New cstate: current_term: 1 leader_uuid: "1965f61888f549f28764af765c122202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c98a06590244652af8b98add9c6e004" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1965f61888f549f28764af765c122202" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 41981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf01d03e0feb446faa6648a235eda07d" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37235 } health_report { overall_health: UNKNOWN } } }
I20260620 21:52:06.088932 12448 raft_consensus.cc:2379] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c98a06590244652af8b98add9c6e004 in current term 1: Already voted for candidate 1965f61888f549f28764af765c122202 in this term.
I20260620 21:52:06.089143 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:06.063395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:47207 (local address 127.0.193.193:41981)
0620 21:52:06.064063 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:06.064069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:06.065366 (+ 1297us) server_negotiation.cc:407] Connection header received
0620 21:52:06.065689 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:06.065692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:06.065731 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:06.065778 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:52:06.066739 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.067651 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.080741 (+ 13090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.081402 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.081543 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:06.081601 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:06.088945 (+ 7344us) server_negotiation.cc:299] Negotiation successful
0620 21:52:06.089033 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583}
I20260620 21:52:06.089148 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:06.061690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:41981 (local address 127.0.193.194:47207)
0620 21:52:06.065282 (+ 3592us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:06.065297 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:06.065451 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:06.066149 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:06.066152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:06.066164 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:52:06.066306 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:06.066313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.079355 (+ 13042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.079358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:06.080622 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:06.080633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.088660 (+ 8027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.088665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:06.088782 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:06.088809 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:52:06.088913 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 21:52:06.088997 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3528,"thread_start_us":135,"threads_started":1}
I20260620 21:52:06.089768 12183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046587af41ab46dbae6e6562e88927a3" candidate_uuid: "8c98a06590244652af8b98add9c6e004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1965f61888f549f28764af765c122202" is_pre_election: true
I20260620 21:52:06.090648 12257 leader_election.cc:302] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c98a06590244652af8b98add9c6e004; no voters: 1965f61888f549f28764af765c122202, cf01d03e0feb446faa6648a235eda07d
I20260620 21:52:06.091089 12527 raft_consensus.cc:3037] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:52:06.092685 12527 raft_consensus.cc:2726] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 21:52:06.112751 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:06.108663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.0.193.193:41981)
0620 21:52:06.108798 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:06.108802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:06.109014 (+ 212us) server_negotiation.cc:407] Connection header received
0620 21:52:06.109172 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:06.109174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:06.109227 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:06.109350 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:52:06.109607 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.110202 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.111261 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:06.111474 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:06.111582 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:06.111913 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:52:06.112349 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:52:06.112483 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:06.112546 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 21:52:06.112602 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:52:06.120874 12322 raft_consensus.cc:1261] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004 [term 1 FOLLOWER]: Refusing update from remote peer 1965f61888f549f28764af765c122202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:52:06.120898 12448 raft_consensus.cc:1261] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d [term 1 FOLLOWER]: Refusing update from remote peer 1965f61888f549f28764af765c122202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:52:06.121479 12526 consensus_queue.cc:1035] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c98a06590244652af8b98add9c6e004" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 35879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:52:06.121913 12539 consensus_queue.cc:1035] T 046587af41ab46dbae6e6562e88927a3 P 1965f61888f549f28764af765c122202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf01d03e0feb446faa6648a235eda07d" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:52:06.129416 12541 mvcc.cc:205] Tried to move back new op lower bound from 7299040567781879808 to 7299040567644192768. Current Snapshot: MvccSnapshot[applied={T|T < 7299040567781879808}]
I20260620 21:52:06.136360 12542 mvcc.cc:205] Tried to move back new op lower bound from 7299040567781879808 to 7299040567644192768. Current Snapshot: MvccSnapshot[applied={T|T < 7299040567781879808}]
I20260620 21:52:06.142355 12018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:52:06.142541 12018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:52:06.145426 12018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4ff7fb0d1c74cab889f8c2135b13d46: Sending DeleteTablet for 3 replicas of tablet 046587af41ab46dbae6e6562e88927a3
I20260620 21:52:06.146481 12422 tablet_service.cc:1514] Processing DeleteTablet for tablet 046587af41ab46dbae6e6562e88927a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:52:06 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:58230
I20260620 21:52:06.147172 12302 tablet_service.cc:1514] Processing DeleteTablet for tablet 046587af41ab46dbae6e6562e88927a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:52:06 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:59346
I20260620 21:52:06.147956 12551 tablet_replica.cc:330] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004: stopping tablet replica
I20260620 21:52:06.148329 12551 raft_consensus.cc:2229] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:52:06.148618 12551 raft_consensus.cc:2258] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:52:06.149199 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 12099
I20260620 21:52:06.149168 12155 tablet_service.cc:1514] Processing DeleteTablet for tablet 046587af41ab46dbae6e6562e88927a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:52:06 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:41166
I20260620 21:52:06.152550 12550 tablet_replica.cc:330] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d: stopping tablet replica
I20260620 21:52:06.152881 12550 raft_consensus.cc:2229] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:52:06.153204 12550 raft_consensus.cc:2258] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:52:06.155589 12551 ts_tablet_manager.cc:1905] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:52:06.155967 12550 ts_tablet_manager.cc:1905] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:52:06.158705 12551 ts_tablet_manager.cc:1918] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:52:06.158906 12551 log.cc:1196] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/046587af41ab46dbae6e6562e88927a3
I20260620 21:52:06.159057 12550 ts_tablet_manager.cc:1918] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:52:06.159436 12551 ts_tablet_manager.cc:1939] T 046587af41ab46dbae6e6562e88927a3 P 8c98a06590244652af8b98add9c6e004: Deleting consensus metadata
I20260620 21:52:06.160914 12005 catalog_manager.cc:4841] TS 8c98a06590244652af8b98add9c6e004 (127.0.193.194:35879): tablet 046587af41ab46dbae6e6562e88927a3 (table test-table [id=f19dc475bc4f4cfd82d0e0d25b635814]) successfully deleted
W20260620 21:52:06.161504 12004 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:41981 (error 108)
I20260620 21:52:06.162168 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 12238
I20260620 21:52:06.162937 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:06.162279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:41981 (local address 127.0.0.1:41184)
0620 21:52:06.162729 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:06.162818 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343}
W20260620 21:52:06.163153 12004 catalog_manager.cc:4584] TS 1965f61888f549f28764af765c122202 (127.0.193.193:41981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 046587af41ab46dbae6e6562e88927a3: Network error: Client connection negotiation failed: client connection to 127.0.193.193:41981: connect: Connection refused (error 111)
I20260620 21:52:06.159159 12550 log.cc:1196] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/046587af41ab46dbae6e6562e88927a3
I20260620 21:52:06.164670 12550 ts_tablet_manager.cc:1939] T 046587af41ab46dbae6e6562e88927a3 P cf01d03e0feb446faa6648a235eda07d: Deleting consensus metadata
I20260620 21:52:06.166390 12002 catalog_manager.cc:4841] TS cf01d03e0feb446faa6648a235eda07d (127.0.193.195:37235): tablet 046587af41ab46dbae6e6562e88927a3 (table test-table [id=f19dc475bc4f4cfd82d0e0d25b635814]) successfully deleted
I20260620 21:52:06.170948 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 12373
I20260620 21:52:06.177428 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 11986
I20260620 21:52:06.186419 775 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 21:52:06.532106 775 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 21:52:06.532274 775 mini_ranger.cc:66] Stopping Ranger...
I20260620 21:52:06.878782 775 mini_ranger.cc:68] Stopped Ranger
2026-06-20 21:52:06.879 UTC [11572] LOG: received smart shutdown request
2026-06-20 21:52:06.882 UTC [11572] LOG: background worker "logical replication launcher" (PID 11581) exited with exit code 1
2026-06-20 21:52:06.883 UTC [11576] LOG: shutting down
2026-06-20 21:52:07.064 UTC [11572] LOG: database system is shut down
2026-06-20T21:52:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50812 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:52:07 dist-test-slave-kk5c krb5kdc[12561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:52:07 dist-test-slave-kk5c krb5kdc[12561](info): set up 2 sockets
Jun 20 21:52:07 dist-test-slave-kk5c krb5kdc[12561](info): commencing operation
krb5kdc: starting...
W20260620 21:52:08.263623 775 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:52:08 dist-test-slave-kk5c krb5kdc[12561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:52:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:52:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/rangeradmin_127.0.193.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/rangeradmin_127.0.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/rangerlookup_127.0.193.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/rangerlookup_127.0.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/HTTP_127.0.193.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/HTTP_127.0.193.212@KRBTEST.COM.keytab.
I20260620 21:52:08.450577 775 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBo3tyv/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/postgres -l logfile start
2026-06-20 21:52:11.372 UTC [12591] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 21:52:11.372 UTC [12591] LOG: listening on IPv4 address "127.0.193.212", port 47601
2026-06-20 21:52:11.376 UTC [12591] LOG: listening on Unix socket "/tmp/.s.PGSQL.47601"
2026-06-20 21:52:11.398 UTC [12594] LOG: database system was shut down at 2026-06-20 21:52:09 UTC
2026-06-20 21:52:11.402 UTC [12591] LOG: database system is ready to accept connections
I20260620 21:52:12.367030 775 mini_postgres.cc:96] Postgres bound to 47601
2026-06-20 21:52:12.373 UTC [12602] FATAL: database "slave" does not exist
127.0.193.212:47601 - accepting connections
I20260620 21:52:12.374147 775 mini_ranger.cc:161] Starting Ranger...
I20260620 21:52:12.389556 775 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 21:52:13.028198 775 mini_ranger.cc:87] Created ranger Postgres database
I20260620 21:52:13.028292 775 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-admin
2026-06-20 21:52:13,311 [I] DB FLAVOR :POSTGRES
2026-06-20 21:52:13,312 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 21:52:13,312 [I] Checking connection..
2026-06-20 21:52:13,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 21:52:13,570 [I] Checking connection passed.
2026-06-20 21:52:13,570 [I] --------- Verifying version history table ---------
2026-06-20 21:52:13,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:52:13,851 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 21:52:13,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:52:14,127 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 21:52:14,127 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 21:52:14,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 21:52:14.380 UTC [12666] WARNING: there is no transaction in progress
2026-06-20 21:52:14,386 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 21:52:14,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 21:52:14,632 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 21:52:14,633 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 21:52:14,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 21:52:14,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 21:52:15,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kk5c.c.gcp-upstream.internal','N') ;"
2026-06-20 21:52:15,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 21:52:15,707 [I] Table x_portal_user does not exist in database ranger
2026-06-20 21:52:15,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 21:52:15,994 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 21:52:15,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 21:52:16,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 21:52:16,547 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 21:52:16,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 21:52:17.695 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.717 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.737 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.756 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.775 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.796 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.863 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.871 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.890 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.902 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.914 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.931 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.942 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.950 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.959 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:17.967 UTC [12774] WARNING: there is no transaction in progress
2026-06-20 21:52:18,891 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 21:52:18,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 21:52:19,161 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 21:52:19,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 21:52:19,426 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 21:52:19,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 21:52:19,694 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 21:52:19,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 21:52:19,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 21:52:20,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kk5c.c.gcp-upstream.internal';"
2026-06-20 21:52:20,445 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 21:52:20,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 21:52:20,719 [I] DB_PATCHES have already been applied
I20260620 21:52:20.725325 775 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBo3tyv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 21:52:20.725435 775 mini_ranger.cc:193] Using host: 127.0.193.212
I20260620 21:52:20.729463 775 mini_ranger.cc:227] Ranger admin URL: http://127.0.193.212:34327
Jun 20, 2026 9:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 9:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 9:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 9:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 9:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 21:52:21.876907 775 mini_ranger.cc:160] Time spent starting Ranger: real 9.503s user 0.000s sys 0.011s
Jun 20, 2026 9:52:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 9:52:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 9:52:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 9:52:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 9:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 9:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 9:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 9:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 9:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 9:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 9:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 9:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 9:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 9:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 9:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 9:52:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 9:52:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 9:52:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 21:52:46.183058 775 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/rangerkms_127.0.193.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/rangerkms_127.0.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.193.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.193.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.193.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/HTTP_127.0.193.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.193.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/HTTP_127.0.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 21:52:46 dist-test-slave-kk5c krb5kdc[12561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992366, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 21:52:46.277932 775 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 21:52:46.289408 775 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 21:52:47.389508 775 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 21:52:47.389587 775 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-kms
I20260620 21:52:47.389601 775 mini_ranger_kms.cc:227] Using postgres at 127.0.193.212:47601
2026-06-20 21:52:47,582 [I] DB FLAVOR :POSTGRES
2026-06-20 21:52:47,582 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 21:52:47,582 [I] Checking connection
2026-06-20 21:52:47,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 21:52:47,868 [I] connection success
2026-06-20 21:52:47,869 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 21:52:47,869 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 21:52:47,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 21:52:48,143 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 21:52:48,143 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 21:52:48,143 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 21:52:48,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.193.212:47601/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 21:52:48,419 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 21:52:48.921815 775 mini_ranger_kms.cc:310] Created kms service
I20260620 21:52:49.116500 775 mini_ranger_kms.cc:326] Created kudu user
I20260620 21:52:49.164645 775 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 21:52:49 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992366, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:49.599095 775 mini_ranger_kms.cc:384] Added ranger policy
I20260620 21:52:49.599295 775 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBo3tyv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBo3tyv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 21:52:49.599321 775 mini_ranger_kms.cc:240] Using host: 127.0.193.212
I20260620 21:52:49.603717 775 mini_ranger_kms.cc:276] Ranger KMS PID: 12955
I20260620 21:52:49.603822 775 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.193.212:47749
Jun 20, 2026 9:52:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 9:52:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 9:52:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 9:52:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 9:52:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 9:52:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 21:52:51.897827 775 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.620s user 0.013s sys 0.163s
I20260620 21:52:51.897979 775 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 21:52:51.898057 775 mini_ranger_kms.cc:401] 127.0.193.212:47749/kms/v1/keys
Jun 20 21:52:52 dist-test-slave-kk5c krb5kdc[12561](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992372, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.193.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 21:52:52 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992372, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.193.212@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
Jun 20 21:52:52 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992372, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.193.212@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
Jun 20 21:52:55 dist-test-slave-kk5c krb5kdc[12561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:52:55.392133 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:38539
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:38539
--ranger_config_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.193.212:47749/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:52:55.534967 13005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:52:55.535388 13005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:52:55.535478 13005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:52:55.539927 13005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:52:55.540019 13005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:52:55.540056 13005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:52:55.540092 13005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:52:55.540132 13005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:52:55.546237 13005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.193.212:47749/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:38539
--ranger_config_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:38539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:52:55.547565 13005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:52:55.548655 13005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:52:55.559175 13010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:55.559290 13005 server_base.cc:1034] running on GCE node
W20260620 21:52:55.559176 13011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:55.559176 13013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:55.560416 13005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:52:55.562033 13005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:52:55.563247 13005 hybrid_clock.cc:648] HybridClock initialized: now 1781992375563205 us; error 63 us; skew 500 ppm
Jun 20 21:52:55 dist-test-slave-kk5c krb5kdc[12561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:52:55.570483 13005 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:52:55.571816 13005 webserver.cc:460] Webserver started at http://127.0.193.254:39081/ using document root <none> and password file <none>
I20260620 21:52:55.572455 13005 fs_manager.cc:359] Metadata directory not provided
I20260620 21:52:55.572520 13005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:52:55.572767 13005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:52:55 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:55.628949 13005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "b21c72b40078451f86a4db7eb1de6120"
format_stamp: "Formatted at 2026-06-20 21:52:55 on dist-test-slave-kk5c"
server_key: "69e89f18330681c6e245a800320bfdd5"
server_key_iv: "8fd7179ada0de7f0ff3a1020a1fdbd2d"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:55.629650 13005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "b21c72b40078451f86a4db7eb1de6120"
format_stamp: "Formatted at 2026-06-20 21:52:55 on dist-test-slave-kk5c"
server_key: "69e89f18330681c6e245a800320bfdd5"
server_key_iv: "8fd7179ada0de7f0ff3a1020a1fdbd2d"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:55.634037 13005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:52:55.649657 13020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:55.651340 13005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:52:55.651503 13005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "b21c72b40078451f86a4db7eb1de6120"
format_stamp: "Formatted at 2026-06-20 21:52:55 on dist-test-slave-kk5c"
server_key: "69e89f18330681c6e245a800320bfdd5"
server_key_iv: "8fd7179ada0de7f0ff3a1020a1fdbd2d"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:55.651641 13005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:52:55.690377 13005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:52:55.693814 13005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:52:55.694034 13005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:52:55.702812 13005 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:38539
I20260620 21:52:55.702812 13072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:38539 every 8 connection(s)
I20260620 21:52:55.704017 13005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:52:55.707700 13073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:52:55.713392 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 13005
I20260620 21:52:55.713550 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:52:55.713869 775 external_mini_cluster.cc:1428] Setting key 43c2b532192cabecc86f822a1821d7ff
I20260620 21:52:55.715355 13073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120: Bootstrap starting.
I20260620 21:52:55.718334 13073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120: Neither blocks nor log segments found. Creating new log.
I20260620 21:52:55.719316 13073 log.cc:824] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120: Log is configured to *not* fsync() on all Append() calls
I20260620 21:52:55.721532 13073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120: No bootstrap required, opened a new log
Jun 20 21:52:55 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:52:55.726315 13073 raft_consensus.cc:348] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21c72b40078451f86a4db7eb1de6120" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38539 } }
I20260620 21:52:55.726662 13073 raft_consensus.cc:374] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:52:55.726768 13073 raft_consensus.cc:729] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21c72b40078451f86a4db7eb1de6120, State: Initialized, Role: FOLLOWER
I20260620 21:52:55.727341 13073 consensus_queue.cc:260] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21c72b40078451f86a4db7eb1de6120" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38539 } }
I20260620 21:52:55.727512 13073 raft_consensus.cc:388] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:52:55.727607 13073 raft_consensus.cc:482] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:52:55.727768 13073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:52:55.728929 13073 raft_consensus.cc:504] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21c72b40078451f86a4db7eb1de6120" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38539 } }
I20260620 21:52:55.729421 13073 leader_election.cc:302] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b21c72b40078451f86a4db7eb1de6120; no voters:
I20260620 21:52:55.729846 13073 leader_election.cc:288] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:52:55.731168 13076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:55.715571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.0.193.254:38539)
0620 21:52:55.716149 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:55.716168 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:55.716205 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:52:55.716902 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:55.716920 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:55.717272 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:55.717634 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:52:55.718584 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:55.719830 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:55.721093 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:55.721772 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:55.721949 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:55.725158 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:52:55.725166 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:52:55.725198 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:52:55.725235 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:52:55.727484 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:55.728123 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:55.728130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:55.728136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:55.728297 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:55.728620 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:55.728623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:55.728625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:55.729220 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:52:55.729414 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:55.729876 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 21:52:55.730195 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":185,"threads_started":1}
I20260620 21:52:55.730454 13079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:52:55.731468 13079 raft_consensus.cc:686] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 1 LEADER]: Becoming Leader. State: Replica: b21c72b40078451f86a4db7eb1de6120, State: Running, Role: LEADER
I20260620 21:52:55.731840 13079 consensus_queue.cc:237] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21c72b40078451f86a4db7eb1de6120" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38539 } }
I20260620 21:52:55.731446 13073 sys_catalog.cc:564] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:52:55.734580 13078 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b21c72b40078451f86a4db7eb1de6120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21c72b40078451f86a4db7eb1de6120" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38539 } } }
I20260620 21:52:55.734740 13078 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [sys.catalog]: This master's current role is: LEADER
I20260620 21:52:55.735100 13080 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b21c72b40078451f86a4db7eb1de6120. Latest consensus state: current_term: 1 leader_uuid: "b21c72b40078451f86a4db7eb1de6120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21c72b40078451f86a4db7eb1de6120" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38539 } } }
I20260620 21:52:55.735265 13080 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [sys.catalog]: This master's current role is: LEADER
I20260620 21:52:55.735157 13083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:52:55.738818 13083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:52:55.739765 13073 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 21:52:55.746330 13083 catalog_manager.cc:1347] Generated new cluster ID: e8e1eb090981438996c26cdb0d2c95f6
I20260620 21:52:55.746415 13083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:52:55.771230 13083 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:52:55.772733 13083 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:52:55.781798 13083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120: Generated new TSK 0
I20260620 21:52:55.782770 13083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:52:56.474954 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.193.212:47749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38539
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:52:56.602742 13118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:52:56.603050 13118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:52:56.603097 13118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:52:56.608280 13118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:52:56.608393 13118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:52:56.608510 13118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:52:56.615445 13118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.193.212:47749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:52:56.616909 13118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:52:56.618618 13118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:52:56.631053 13124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:56.631443 13118 server_base.cc:1034] running on GCE node
W20260620 21:52:56.631582 13123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:56.631831 13126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:56.632299 13118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:52:56.633044 13118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:52:56.634208 13118 hybrid_clock.cc:648] HybridClock initialized: now 1781992376634173 us; error 56 us; skew 500 ppm
Jun 20 21:52:56 dist-test-slave-kk5c krb5kdc[12561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:52:56.637455 13118 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:52:56.638427 13118 webserver.cc:460] Webserver started at http://127.0.193.193:33531/ using document root <none> and password file <none>
I20260620 21:52:56.639029 13118 fs_manager.cc:359] Metadata directory not provided
I20260620 21:52:56.639124 13118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:52:56.639392 13118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:52:56 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:56.658993 13118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "755163dedcda4177ad0fdd4ddeb61844"
format_stamp: "Formatted at 2026-06-20 21:52:56 on dist-test-slave-kk5c"
server_key: "4f5b1e350dcee0b19393e3343bb96fd3"
server_key_iv: "21460ea2792e0fa68169113e099f9703"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:56.660022 13118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "755163dedcda4177ad0fdd4ddeb61844"
format_stamp: "Formatted at 2026-06-20 21:52:56 on dist-test-slave-kk5c"
server_key: "4f5b1e350dcee0b19393e3343bb96fd3"
server_key_iv: "21460ea2792e0fa68169113e099f9703"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:56.666977 13118 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 21:52:56.686509 13135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:56.688412 13118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:52:56.688642 13118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "755163dedcda4177ad0fdd4ddeb61844"
format_stamp: "Formatted at 2026-06-20 21:52:56 on dist-test-slave-kk5c"
server_key: "4f5b1e350dcee0b19393e3343bb96fd3"
server_key_iv: "21460ea2792e0fa68169113e099f9703"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:56.688886 13118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:52:56.711666 13118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:52:56.715443 13118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:52:56.715719 13118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:52:56.716480 13118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:52:56.717818 13118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:52:56.717924 13118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:56.718045 13118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:52:56.718123 13118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:56.732185 13118 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:39055
I20260620 21:52:56.733460 13118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:52:56.733758 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 13118
I20260620 21:52:56.733901 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:52:56.734269 775 external_mini_cluster.cc:1428] Setting key 6571341f27e4ca9bb9b9c91e119345f9
I20260620 21:52:56.735004 13248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:39055 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
Jun 20 21:52:56 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:52:56.777748 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:56.743416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:54943 (local address 127.0.193.254:38539)
0620 21:52:56.759462 (+ 16046us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:56.759477 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:56.759501 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:52:56.759563 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:56.759567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:56.759635 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:56.761624 (+ 1989us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:52:56.761782 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:56.762755 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:56.764834 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:56.765260 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:56.766185 (+ 925us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:56.771894 (+ 5709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:52:56.771900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:52:56.771903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:52:56.771941 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:52:56.774314 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:56.775328 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:56.775334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:56.775337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:56.775456 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:56.775992 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:56.775997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:56.775998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:56.776279 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:52:56.776382 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:56.777248 (+ 866us) server_negotiation.cc:299] Negotiation successful
0620 21:52:56.777422 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15926,"thread_start_us":106,"threads_started":1}
I20260620 21:52:56.781157 13252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:56.742092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:38539 (local address 127.0.193.193:54943)
0620 21:52:56.751667 (+ 9575us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:56.751759 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:56.753007 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:56.760125 (+ 7118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:56.760139 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:56.760557 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:52:56.761418 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:56.761444 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:56.762952 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:56.762956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:56.764612 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:56.764628 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:56.765477 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:56.765482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:56.765731 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:56.766602 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:52:56.766628 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:52:56.771568 (+ 4940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:52:56.774549 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:56.774557 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:56.774580 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:56.775112 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:56.775596 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:56.775601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:56.775604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:56.775846 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:56.776472 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:52:56.776483 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:52:56.776923 (+ 440us) client_negotiation.cc:769] Sending connection context
0620 21:52:56.778377 (+ 1454us) client_negotiation.cc:240] Negotiation successful
0620 21:52:56.778836 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9258,"thread_start_us":187,"threads_started":1}
I20260620 21:52:56.782724 13249 heartbeater.cc:347] Connected to a master server at 127.0.193.254:38539
I20260620 21:52:56.783030 13249 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:56.783806 13249 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
I20260620 21:52:56.786078 13037 ts_manager.cc:194] Registered new tserver with Master: 755163dedcda4177ad0fdd4ddeb61844 (127.0.193.193:39055)
I20260620 21:52:56.788216 13037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:54943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:52:56.809556 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.193.212:47749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38539
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:52:56 dist-test-slave-kk5c krb5kdc[12561](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 21:52:56.957804 13257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:52:56.958220 13257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:52:56.958334 13257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:52:56.962888 13257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:52:56.962987 13257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:52:56.963091 13257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:52:56.968871 13257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.193.212:47749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:52:56.970086 13257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:52:56.971138 13257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:52:56.981145 13266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:56.981532 13263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:56.981608 13257 server_base.cc:1034] running on GCE node
W20260620 21:52:56.981808 13264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:56.982250 13257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:52:56.983012 13257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:52:56.984223 13257 hybrid_clock.cc:648] HybridClock initialized: now 1781992376984185 us; error 56 us; skew 500 ppm
Jun 20 21:52:56 dist-test-slave-kk5c krb5kdc[12561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:52:56.987426 13257 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:52:56.988705 13257 webserver.cc:460] Webserver started at http://127.0.193.194:40583/ using document root <none> and password file <none>
I20260620 21:52:56.989432 13257 fs_manager.cc:359] Metadata directory not provided
I20260620 21:52:56.989522 13257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:52:56.989766 13257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:52:56 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:57.010578 13257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "6559e16b69404159a9936d466c797b18"
format_stamp: "Formatted at 2026-06-20 21:52:57 on dist-test-slave-kk5c"
server_key: "fc86ee1b3b1b3c66506e5e315e0f5061"
server_key_iv: "32e71260577c2cc6bf52debeff822bb1"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:57.011406 13257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "6559e16b69404159a9936d466c797b18"
format_stamp: "Formatted at 2026-06-20 21:52:57 on dist-test-slave-kk5c"
server_key: "fc86ee1b3b1b3c66506e5e315e0f5061"
server_key_iv: "32e71260577c2cc6bf52debeff822bb1"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:57.015851 13257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:52:57.030344 13273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:57.031836 13257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:52:57.031985 13257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "6559e16b69404159a9936d466c797b18"
format_stamp: "Formatted at 2026-06-20 21:52:57 on dist-test-slave-kk5c"
server_key: "fc86ee1b3b1b3c66506e5e315e0f5061"
server_key_iv: "32e71260577c2cc6bf52debeff822bb1"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:57.032140 13257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:52:57.062383 13257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:52:57.067031 13257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:52:57.067298 13257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:52:57.067977 13257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:52:57.068902 13257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:52:57.068971 13257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:57.069038 13257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:52:57.069094 13257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:57.079447 13257 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:40175
I20260620 21:52:57.080590 13257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:52:57.082767 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 13257
I20260620 21:52:57.082901 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:52:57.083266 775 external_mini_cluster.cc:1428] Setting key d6acc4311131164c7a44741b74257a4b
I20260620 21:52:57.093479 13386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:40175 every 8 connection(s)
Jun 20 21:52:57 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
I20260620 21:52:57.100483 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:57.082448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:47913 (local address 127.0.193.254:38539)
0620 21:52:57.083715 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:57.083720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:57.084440 (+ 720us) server_negotiation.cc:407] Connection header received
0620 21:52:57.086025 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:57.086031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:57.086093 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:57.086150 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:52:57.088064 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:57.088951 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:57.090883 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:57.091347 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:57.091448 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:57.095020 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:52:57.095028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:52:57.095033 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:52:57.095065 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:52:57.097105 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:57.097866 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:57.097869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:57.097871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:57.097964 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:57.098514 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:57.098520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:57.098525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:57.098796 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:52:57.098885 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:57.100008 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0620 21:52:57.100186 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1166}
I20260620 21:52:57.102707 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:57.082946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:38539 (local address 127.0.193.194:47913)
0620 21:52:57.084293 (+ 1347us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:57.084348 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:57.085759 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:57.086646 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:57.086660 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:57.087060 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:52:57.087869 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:57.087892 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:57.089120 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:57.089127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:57.090715 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:57.090730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:57.091774 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:57.091781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:57.092034 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:57.092823 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:52:57.092847 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:52:57.094782 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:52:57.097263 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:57.097271 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:57.097299 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:57.097718 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:57.098066 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:57.098071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:57.098072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:57.098321 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:57.098979 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:52:57.098990 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:52:57.099673 (+ 683us) client_negotiation.cc:769] Sending connection context
0620 21:52:57.100993 (+ 1320us) client_negotiation.cc:240] Negotiation successful
0620 21:52:57.101444 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"thread_start_us":183,"threads_started":1}
I20260620 21:52:57.104475 13387 heartbeater.cc:347] Connected to a master server at 127.0.193.254:38539
I20260620 21:52:57.104736 13387 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:57.105381 13387 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
I20260620 21:52:57.106777 13037 ts_manager.cc:194] Registered new tserver with Master: 6559e16b69404159a9936d466c797b18 (127.0.193.194:40175)
I20260620 21:52:57.107618 13037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:47913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:52:57.123940 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.193.212:47749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38539
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:52:57.272290 13392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:52:57.272708 13392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:52:57.272783 13392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:52:57.278005 13392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:52:57.278107 13392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:52:57.278210 13392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:52:57.284832 13392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.193.212:47749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:52:57.286108 13392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:52:57.287371 13392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:52:57.299729 13398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:57.299729 13400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:57.300283 13397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:57.301226 13392 server_base.cc:1034] running on GCE node
I20260620 21:52:57.301753 13392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:52:57.302711 13392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:52:57.310194 13392 hybrid_clock.cc:648] HybridClock initialized: now 1781992377310096 us; error 122 us; skew 500 ppm
Jun 20 21:52:57 dist-test-slave-kk5c krb5kdc[12561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:52:57.317919 13392 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:52:57.319438 13392 webserver.cc:460] Webserver started at http://127.0.193.195:35529/ using document root <none> and password file <none>
I20260620 21:52:57.320326 13392 fs_manager.cc:359] Metadata directory not provided
I20260620 21:52:57.320427 13392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:52:57.320673 13392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 21:52:57 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:57.351796 13392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "42ec5a99259c4d4691738cecba4fee02"
format_stamp: "Formatted at 2026-06-20 21:52:57 on dist-test-slave-kk5c"
server_key: "2819f5887186fffcacb5aa00057847bb"
server_key_iv: "a4d4cf40575c0ff3912acb4131dadc6b"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:57.352568 13392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "42ec5a99259c4d4691738cecba4fee02"
format_stamp: "Formatted at 2026-06-20 21:52:57 on dist-test-slave-kk5c"
server_key: "2819f5887186fffcacb5aa00057847bb"
server_key_iv: "a4d4cf40575c0ff3912acb4131dadc6b"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:57.360549 13392 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260620 21:52:57.388247 13407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:57.390416 13392 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 21:52:57.390590 13392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "42ec5a99259c4d4691738cecba4fee02"
format_stamp: "Formatted at 2026-06-20 21:52:57 on dist-test-slave-kk5c"
server_key: "2819f5887186fffcacb5aa00057847bb"
server_key_iv: "a4d4cf40575c0ff3912acb4131dadc6b"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:57.390750 13392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 21:52:57 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:57.412128 13392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:52:57.416021 13392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:52:57.416236 13392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:52:57.416987 13392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:52:57.418066 13392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:52:57.418133 13392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:57.418208 13392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:52:57.418241 13392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:57.434155 13392 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:33095
I20260620 21:52:57.435667 13392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:52:57.436734 13520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:33095 every 8 connection(s)
I20260620 21:52:57.443773 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 13392
I20260620 21:52:57.443902 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:52:57.444209 775 external_mini_cluster.cc:1428] Setting key 0233dfa25bacd5d6869f802a2f526d91
Jun 20 21:52:57 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:52:57.456724 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:57.437903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:40221 (local address 127.0.193.254:38539)
0620 21:52:57.438035 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:57.438039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:57.442508 (+ 4469us) server_negotiation.cc:407] Connection header received
0620 21:52:57.443866 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:57.443869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:57.443936 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:57.443996 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:52:57.445702 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:57.446461 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:57.448108 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:57.448462 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:57.448545 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:57.452033 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:52:57.452038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:52:57.452041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:52:57.452071 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:52:57.453828 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:57.454634 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:57.454638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:57.454640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:57.454728 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:57.455173 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:57.455176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:57.455178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:57.455440 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:52:57.455526 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:57.456374 (+ 848us) server_negotiation.cc:299] Negotiation successful
0620 21:52:57.456521 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:52:57.457785 13523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:57.438305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:38539 (local address 127.0.193.195:40221)
0620 21:52:57.442327 (+ 4022us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:57.442402 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:57.443589 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:57.444445 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:57.444455 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:57.444813 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:52:57.445528 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:57.445549 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:57.446622 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:57.446625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:57.447967 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:57.447977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:57.449039 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:57.449043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:57.449287 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:57.450055 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:52:57.450077 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:52:57.451818 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:52:57.453989 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:57.453996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:57.454015 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:57.454514 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:57.454833 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:57.454836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:57.454838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:57.455073 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:57.455619 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:52:57.455625 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:52:57.455984 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 21:52:57.456305 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 21:52:57.456700 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3732,"thread_start_us":132,"threads_started":1}
I20260620 21:52:57.459133 13521 heartbeater.cc:347] Connected to a master server at 127.0.193.254:38539
I20260620 21:52:57.459414 13521 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:57.460001 13521 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
I20260620 21:52:57.461103 13037 ts_manager.cc:194] Registered new tserver with Master: 42ec5a99259c4d4691738cecba4fee02 (127.0.193.195:33095)
I20260620 21:52:57.461714 13037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:40221
I20260620 21:52:57.473147 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:52:57.474032 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 13118
I20260620 21:52:57.482981 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 13257
I20260620 21:52:57.489881 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 13392
I20260620 21:52:57.496754 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 13005
I20260620 21:52:57.509791 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:38539
--webserver_interface=127.0.193.254
--webserver_port=39081
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:38539
--ranger_config_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.193.212:47749/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.193.212:47749/kms with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:52:57.647130 13524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:52:57.647454 13524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:52:57.647496 13524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:52:57.651793 13524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:52:57.651877 13524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:52:57.651897 13524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:52:57.651914 13524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:52:57.651930 13524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:52:57.658041 13524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.193.212:47749/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:38539
--ranger_config_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:38539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=39081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:52:57.659168 13524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:52:57.660349 13524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:52:57.669409 13530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:57.669410 13532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:57.669431 13529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:57.669590 13524 server_base.cc:1034] running on GCE node
I20260620 21:52:57.670320 13524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:52:57.671619 13524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:52:57.672833 13524 hybrid_clock.cc:648] HybridClock initialized: now 1781992377672818 us; error 31 us; skew 500 ppm
Jun 20 21:52:57 dist-test-slave-kk5c krb5kdc[12561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:52:57.676007 13524 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:52:57.677021 13524 webserver.cc:460] Webserver started at http://127.0.193.254:39081/ using document root <none> and password file <none>
I20260620 21:52:57.677621 13524 fs_manager.cc:359] Metadata directory not provided
I20260620 21:52:57.677729 13524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 21:52:57 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:57.765084 13524 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:52:57.768199 13540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:57.769871 13524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:52:57.770051 13524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "b21c72b40078451f86a4db7eb1de6120"
format_stamp: "Formatted at 2026-06-20 21:52:55 on dist-test-slave-kk5c"
server_key: "69e89f18330681c6e245a800320bfdd5"
server_key_iv: "8fd7179ada0de7f0ff3a1020a1fdbd2d"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:57.770839 13524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:52:57.786834 13524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:52:57.790278 13524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:52:57.790458 13524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:52:57.798593 13524 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:38539
I20260620 21:52:57.798620 13592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:38539 every 8 connection(s)
I20260620 21:52:57.799933 13524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:52:57.800734 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 13524
I20260620 21:52:57.801216 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.193.212:47749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:39055
--local_ip_for_outbound_sockets=127.0.193.193
--tserver_master_addrs=127.0.193.254:38539
--webserver_port=33531
--webserver_interface=127.0.193.193
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.193.212:47749/kms with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:52:57.805806 13593 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:52:57.807742 13593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120: Bootstrap starting.
I20260620 21:52:57.825904 13593 log.cc:824] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120: Log is configured to *not* fsync() on all Append() calls
I20260620 21:52:57.833169 13593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:52:57.833658 13593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120: Bootstrap complete.
I20260620 21:52:57.838272 13593 raft_consensus.cc:348] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21c72b40078451f86a4db7eb1de6120" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38539 } }
I20260620 21:52:57.838709 13593 raft_consensus.cc:729] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21c72b40078451f86a4db7eb1de6120, State: Initialized, Role: FOLLOWER
I20260620 21:52:57.839311 13593 consensus_queue.cc:260] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21c72b40078451f86a4db7eb1de6120" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38539 } }
I20260620 21:52:57.839444 13593 raft_consensus.cc:388] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:52:57.839540 13593 raft_consensus.cc:482] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:52:57.839672 13593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:52:57.841843 13593 raft_consensus.cc:504] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21c72b40078451f86a4db7eb1de6120" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38539 } }
I20260620 21:52:57.842240 13593 leader_election.cc:302] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b21c72b40078451f86a4db7eb1de6120; no voters:
I20260620 21:52:57.842639 13593 leader_election.cc:288] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 21:52:57.842736 13597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:52:57.843009 13597 raft_consensus.cc:686] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [term 2 LEADER]: Becoming Leader. State: Replica: b21c72b40078451f86a4db7eb1de6120, State: Running, Role: LEADER
I20260620 21:52:57.843429 13597 consensus_queue.cc:237] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21c72b40078451f86a4db7eb1de6120" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38539 } }
I20260620 21:52:57.844039 13593 sys_catalog.cc:564] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:52:57.845234 13599 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b21c72b40078451f86a4db7eb1de6120. Latest consensus state: current_term: 2 leader_uuid: "b21c72b40078451f86a4db7eb1de6120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21c72b40078451f86a4db7eb1de6120" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38539 } } }
I20260620 21:52:57.845369 13599 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [sys.catalog]: This master's current role is: LEADER
I20260620 21:52:57.845330 13598 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b21c72b40078451f86a4db7eb1de6120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21c72b40078451f86a4db7eb1de6120" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38539 } } }
I20260620 21:52:57.845441 13598 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120 [sys.catalog]: This master's current role is: LEADER
I20260620 21:52:57.845863 13604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:52:57.849529 13604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:52:57.850169 13593 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 21:52:57.850251 13604 catalog_manager.cc:1259] Loaded cluster ID: e8e1eb090981438996c26cdb0d2c95f6
I20260620 21:52:57.850306 13604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:52:57.852948 13604 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:52:57.853693 13604 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120: Loaded TSK: 0
I20260620 21:52:57.855054 13604 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:52:57.951489 13595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:52:57.951802 13595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:52:57.951892 13595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:52:57.957270 13595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:52:57.957371 13595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:52:57.957484 13595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:52:57.963922 13595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.193.212:47749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:39055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=33531
--tserver_master_addrs=127.0.193.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:52:57.965066 13595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:52:57.966115 13595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:52:57.978376 13630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:57.978619 13631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:57.978459 13633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:57.978849 13595 server_base.cc:1034] running on GCE node
I20260620 21:52:57.979564 13595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:52:57.980372 13595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:52:57.981556 13595 hybrid_clock.cc:648] HybridClock initialized: now 1781992377981526 us; error 41 us; skew 500 ppm
Jun 20 21:52:57 dist-test-slave-kk5c krb5kdc[12561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:52:57.984611 13595 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:52:57.985817 13595 webserver.cc:460] Webserver started at http://127.0.193.193:33531/ using document root <none> and password file <none>
I20260620 21:52:57.986390 13595 fs_manager.cc:359] Metadata directory not provided
I20260620 21:52:57.986495 13595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 21:52:57 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:58.016916 13595 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 21:52:58.020417 13641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:58.022167 13595 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 21:52:58.022316 13595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "755163dedcda4177ad0fdd4ddeb61844"
format_stamp: "Formatted at 2026-06-20 21:52:56 on dist-test-slave-kk5c"
server_key: "4f5b1e350dcee0b19393e3343bb96fd3"
server_key_iv: "21460ea2792e0fa68169113e099f9703"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:58.023077 13595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:52:58.039929 13595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:52:58.044596 13595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:52:58.044934 13595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:52:58.045789 13595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:52:58.046908 13595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:52:58.046980 13595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:58.047039 13595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:52:58.047061 13595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:58.058651 13595 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:39055
I20260620 21:52:58.059311 13754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:39055 every 8 connection(s)
I20260620 21:52:58.060060 13595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:52:58.069526 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 13595
I20260620 21:52:58.070073 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.193.212:47749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:40175
--local_ip_for_outbound_sockets=127.0.193.194
--tserver_master_addrs=127.0.193.254:38539
--webserver_port=40583
--webserver_interface=127.0.193.194
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.193.212:47749/kms with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:52:58 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:52:58.085206 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:58.063117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:38539 (local address 127.0.193.193:35949)
0620 21:52:58.065535 (+ 2418us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:58.065603 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:58.066722 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:58.068387 (+ 1665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:58.068396 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:58.068774 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:52:58.069455 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:58.069474 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:58.071478 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:58.071482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:58.073251 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:58.073267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:58.074707 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:58.074711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:58.074934 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:58.075934 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:52:58.075957 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:52:58.078226 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:52:58.081215 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:58.081222 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:58.081240 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:58.081630 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:58.082083 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:58.082086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:58.082088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:58.082287 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:58.083135 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:52:58.083141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:52:58.083477 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 21:52:58.083687 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 21:52:58.084083 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2163,"thread_start_us":142,"threads_started":1}
I20260620 21:52:58.087358 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:58.063016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:35949 (local address 127.0.193.254:38539)
0620 21:52:58.065305 (+ 2289us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:58.065329 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:58.065749 (+ 420us) server_negotiation.cc:407] Connection header received
0620 21:52:58.067342 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:58.067358 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:58.067844 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:58.068259 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:52:58.069670 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:58.071293 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:58.073428 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:58.074257 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:58.074445 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:58.078529 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:52:58.078537 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:52:58.078558 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:52:58.078596 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:52:58.081044 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:58.081801 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:58.081808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:58.081814 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:58.081965 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:58.082406 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:58.082409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:58.082411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:58.082942 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:52:58.083065 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:58.085854 (+ 2789us) server_negotiation.cc:299] Negotiation successful
0620 21:52:58.086227 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2073,"thread_start_us":134,"threads_started":1}
I20260620 21:52:58.090186 13755 heartbeater.cc:347] Connected to a master server at 127.0.193.254:38539
I20260620 21:52:58.090495 13755 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:58.091168 13755 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
W20260620 21:52:58.093854 13755 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:38539 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 21:52:58.094017 13755 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:58.094241 13755 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
I20260620 21:52:58.095296 13755 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:58.095511 13755 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
W20260620 21:52:58.096334 13755 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:52:58.242909 13759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:52:58.243430 13759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:52:58.243512 13759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:52:58.250993 13759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:52:58.251106 13759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:52:58.251246 13759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:52:58.259658 13759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.193.212:47749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:40175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=40583
--tserver_master_addrs=127.0.193.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:52:58.261368 13759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:52:58.262744 13759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:52:58.277616 13765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:58.277606 13767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:58.277631 13764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:58.279006 13759 server_base.cc:1034] running on GCE node
I20260620 21:52:58.279672 13759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:52:58.280561 13759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:52:58.283885 13759 hybrid_clock.cc:648] HybridClock initialized: now 1781992378283840 us; error 49 us; skew 500 ppm
Jun 20 21:52:58 dist-test-slave-kk5c krb5kdc[12561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:52:58.287945 13759 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:52:58.289454 13759 webserver.cc:460] Webserver started at http://127.0.193.194:40583/ using document root <none> and password file <none>
I20260620 21:52:58.290298 13759 fs_manager.cc:359] Metadata directory not provided
I20260620 21:52:58.290410 13759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 21:52:58 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:58.321753 13759 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:52:58.328215 13775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:58.330417 13759 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260620 21:52:58.330595 13759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "6559e16b69404159a9936d466c797b18"
format_stamp: "Formatted at 2026-06-20 21:52:57 on dist-test-slave-kk5c"
server_key: "fc86ee1b3b1b3c66506e5e315e0f5061"
server_key_iv: "32e71260577c2cc6bf52debeff822bb1"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:58.331486 13759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:52:58.363986 13759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:52:58.367998 13759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:52:58.368230 13759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:52:58.369011 13759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:52:58.370128 13759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:52:58.370193 13759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:58.370249 13759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:52:58.370282 13759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:58.384016 13759 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:40175
I20260620 21:52:58.384088 13888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:40175 every 8 connection(s)
I20260620 21:52:58.385474 13759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:52:58.386897 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 13759
I20260620 21:52:58.387418 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.193.212:47749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:33095
--local_ip_for_outbound_sockets=127.0.193.195
--tserver_master_addrs=127.0.193.254:38539
--webserver_port=35529
--webserver_interface=127.0.193.195
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.193.212:47749/kms with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:52:58 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:52:58.419879 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:58.390553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:47741 (local address 127.0.193.254:38539)
0620 21:52:58.390693 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:58.390698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:58.392597 (+ 1899us) server_negotiation.cc:407] Connection header received
0620 21:52:58.394032 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:58.394038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:58.394104 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:58.394161 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:52:58.396086 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:58.396954 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:58.400730 (+ 3776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:58.401123 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:58.401244 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:58.413298 (+ 12054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:52:58.413305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:52:58.413308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:52:58.413345 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:52:58.415177 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:58.416031 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:58.416035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:58.416037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:58.416136 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:58.416653 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:58.416656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:58.416658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:58.416955 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:52:58.417066 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:58.419555 (+ 2489us) server_negotiation.cc:299] Negotiation successful
0620 21:52:58.419724 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:52:58.419343 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:58.391037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:38539 (local address 127.0.193.194:47741)
0620 21:52:58.392387 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:58.392457 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:58.393722 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:58.394660 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:58.394671 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:58.395072 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:52:58.395870 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:58.395894 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:58.397141 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:58.397145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:58.398683 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:58.398700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:58.401529 (+ 2829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:58.401533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:58.401794 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:58.411009 (+ 9215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:52:58.411046 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:52:58.412971 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:52:58.415405 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:58.415413 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:58.415436 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:58.415884 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:58.416265 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:58.416269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:58.416271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:58.416500 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:58.417129 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:52:58.417136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:52:58.417530 (+ 394us) client_negotiation.cc:769] Sending connection context
0620 21:52:58.417756 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 21:52:58.418142 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"thread_start_us":200,"threads_started":1}
I20260620 21:52:58.421710 13889 heartbeater.cc:347] Connected to a master server at 127.0.193.254:38539
I20260620 21:52:58.421981 13889 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:58.422860 13889 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
W20260620 21:52:58.424618 13889 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:38539 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 21:52:58.424758 13889 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:58.424948 13889 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
I20260620 21:52:58.425794 13889 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:58.425978 13889 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
W20260620 21:52:58.426777 13889 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:52:58.583832 13891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:52:58.584210 13891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:52:58.584288 13891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:52:58.590646 13891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:52:58.590744 13891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:52:58.590850 13891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:52:58.598498 13891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.193.212:47749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:33095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=35529
--tserver_master_addrs=127.0.193.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:52:58.600039 13891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:52:58.601485 13891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:52:58.612773 13891 server_base.cc:1034] running on GCE node
W20260620 21:52:58.612723 13899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:58.612723 13898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:52:58.612758 13901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:52:58.613499 13891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:52:58.614210 13891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:52:58.615417 13891 hybrid_clock.cc:648] HybridClock initialized: now 1781992378615380 us; error 58 us; skew 500 ppm
Jun 20 21:52:58 dist-test-slave-kk5c krb5kdc[12561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:52:58.618996 13891 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:52:58.620461 13891 webserver.cc:460] Webserver started at http://127.0.193.195:35529/ using document root <none> and password file <none>
I20260620 21:52:58.621282 13891 fs_manager.cc:359] Metadata directory not provided
I20260620 21:52:58.621402 13891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 21:52:58 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:58.640350 13891 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 21:52:58.643276 13909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:58.644820 13891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:52:58.644963 13891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "42ec5a99259c4d4691738cecba4fee02"
format_stamp: "Formatted at 2026-06-20 21:52:57 on dist-test-slave-kk5c"
server_key: "2819f5887186fffcacb5aa00057847bb"
server_key_iv: "a4d4cf40575c0ff3912acb4131dadc6b"
server_key_version: "kuduclusterkey@0"
I20260620 21:52:58.645550 13891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:52:58.666787 13891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:52:58.670599 13891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:52:58.670823 13891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:52:58.671643 13891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:52:58.672711 13891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:52:58.672796 13891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:58.672869 13891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:52:58.672927 13891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:52:58.689123 13891 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:33095
I20260620 21:52:58.690694 13891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:52:58.691919 14022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:33095 every 8 connection(s)
I20260620 21:52:58.695990 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 13891
Jun 20 21:52:58 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:52:58.710572 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:58.696710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.0.193.254:38539)
0620 21:52:58.698008 (+ 1298us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:58.698021 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:58.698037 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:52:58.698080 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:58.698083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:58.698127 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:58.698237 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:52:58.699756 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:58.700584 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:58.701781 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:58.702114 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:58.702231 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:58.705100 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:52:58.705105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:52:58.705107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:52:58.705137 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:52:58.707432 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:58.708432 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:58.708435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:58.708437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:58.708526 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:58.709125 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:58.709127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:58.709129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:58.709363 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:52:58.709539 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:58.710175 (+ 636us) server_negotiation.cc:299] Negotiation successful
0620 21:52:58.710362 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":1219,"thread_start_us":112,"threads_started":1}
I20260620 21:52:58.710606 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:58.693944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:49485 (local address 127.0.193.254:38539)
0620 21:52:58.694090 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:58.694095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:58.695325 (+ 1230us) server_negotiation.cc:407] Connection header received
0620 21:52:58.697358 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:58.697362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:58.697424 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:58.697511 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:52:58.699624 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:58.700516 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:58.702246 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:58.702682 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:58.702796 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:58.706538 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:52:58.706542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:52:58.706547 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:52:58.706598 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:52:58.708302 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:58.708930 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:58.708933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:58.708935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:58.709012 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:58.709456 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:58.709460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:58.709462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:58.709702 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:52:58.709805 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:58.710398 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 21:52:58.710499 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":60}
I20260620 21:52:58.712222 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:58.694506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:38539 (local address 127.0.193.195:49485)
0620 21:52:58.695058 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:58.695117 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:58.697112 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:58.697882 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:58.697894 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:58.698309 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:52:58.699162 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:58.699433 (+ 271us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:58.700690 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:58.700693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:58.702115 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:58.702125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:58.702983 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:58.702988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:58.703243 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:58.704195 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:52:58.704234 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:52:58.706350 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:52:58.708432 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:58.708439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:58.708459 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:58.708813 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:58.709107 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:52:58.709112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:52:58.709115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:52:58.709311 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:52:58.709813 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:52:58.709819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:52:58.710108 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 21:52:58.710667 (+ 559us) client_negotiation.cc:240] Negotiation successful
0620 21:52:58.711045 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":182,"threads_started":1}
I20260620 21:52:58.713413 14023 heartbeater.cc:347] Connected to a master server at 127.0.193.254:38539
I20260620 21:52:58.713671 14023 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:58.714250 14023 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
W20260620 21:52:58.716135 14023 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:38539 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 21:52:58.716287 14023 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:58.716516 14023 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
I20260620 21:52:58.717521 14023 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:58.717762 14023 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
W20260620 21:52:58.718613 14023 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 21:52:59.096657 13755 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:59.097008 13755 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
I20260620 21:52:59.098322 13557 ts_manager.cc:194] Registered new tserver with Master: 755163dedcda4177ad0fdd4ddeb61844 (127.0.193.193:39055)
I20260620 21:52:59.100095 13557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:35949
Jun 20 21:52:59 dist-test-slave-kk5c krb5kdc[12561](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781992379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:52:59.427004 13889 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:59.427368 13889 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
I20260620 21:52:59.428283 13557 ts_manager.cc:194] Registered new tserver with Master: 6559e16b69404159a9936d466c797b18 (127.0.193.194:40175)
I20260620 21:52:59.429020 13557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:47741
Jun 20 21:52:59 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
I20260620 21:52:59.718901 14023 heartbeater.cc:464] Registering TS with master...
I20260620 21:52:59.719274 14023 heartbeater.cc:511] Master 127.0.193.254:38539 requested a full tablet report, sending...
I20260620 21:52:59.720166 13557 ts_manager.cc:194] Registered new tserver with Master: 42ec5a99259c4d4691738cecba4fee02 (127.0.193.195:33095)
I20260620 21:52:59.720939 13557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:49485
I20260620 21:52:59.726107 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:52:59.737874 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:59.727760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.0.193.254:38539)
0620 21:52:59.728079 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:59.728084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:59.728423 (+ 339us) server_negotiation.cc:407] Connection header received
0620 21:52:59.728577 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:59.728580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:59.728641 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:59.728733 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:52:59.729662 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:59.730503 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:59.731720 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:59.732081 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:59.732196 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:59.733927 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:52:59.733931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:52:59.733934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:52:59.733962 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:52:59.735885 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:59.736439 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:59.736442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:59.736444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:59.736551 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:52:59.736877 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:52:59.736881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:52:59.736882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:52:59.737125 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:52:59.737262 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:59.737552 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 21:52:59.737691 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":106,"threads_started":1}
I20260620 21:52:59.742180 13557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:52:59.747005 13557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:52:59.768028 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:59.759255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:40175 (local address 127.0.0.1:54628)
0620 21:52:59.761184 (+ 1929us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:59.761255 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:59.761418 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:59.762402 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:59.762409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:59.762432 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:52:59.762566 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:59.762573 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:59.764032 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:59.764036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:59.765354 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:59.765365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:59.767491 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:59.767494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:59.767604 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:59.767653 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 21:52:59.767758 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 21:52:59.767875 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1702,"mutex_wait_us":1,"thread_start_us":131,"threads_started":1}
I20260620 21:52:59.768592 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:59.759359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.0.193.194:40175)
0620 21:52:59.761301 (+ 1942us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:59.761308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:59.761335 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:52:59.761642 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:59.761648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:59.761869 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:59.762021 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:52:59.762714 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:59.763901 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:59.765493 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:59.766275 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:59.766375 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:59.766517 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:59.768355 (+ 1838us) server_negotiation.cc:299] Negotiation successful
0620 21:52:59.768463 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1826,"thread_start_us":80,"threads_started":1}
I20260620 21:52:59.769256 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:59.759783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:33095 (local address 127.0.0.1:35826)
0620 21:52:59.761689 (+ 1906us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:59.761703 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:59.761864 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:59.762546 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:59.762549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:59.762570 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:52:59.762663 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:59.762670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:59.764973 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:59.764976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:59.766149 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:59.766161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:59.768828 (+ 2667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:59.768835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:59.768929 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:59.768954 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:52:59.769045 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 21:52:59.769129 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1843,"thread_start_us":93,"threads_started":1}
I20260620 21:52:59.769759 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:59.759456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35826 (local address 127.0.193.195:33095)
0620 21:52:59.761547 (+ 2091us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:59.761555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:59.761791 (+ 236us) server_negotiation.cc:407] Connection header received
0620 21:52:59.762032 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:59.762038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:59.762231 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:59.762358 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:52:59.762804 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:59.764822 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:59.766305 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:59.767988 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:59.768082 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:59.768536 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:59.769520 (+ 984us) server_negotiation.cc:299] Negotiation successful
0620 21:52:59.769625 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1984,"thread_start_us":99,"threads_started":1}
I20260620 21:52:59.772001 13823 tablet_service.cc:1467] Processing CreateTablet for tablet d929df5ce8c14391af1eeb4c6c611f22 (DEFAULT_TABLE table=test-table [id=18596b7f911e458f9aedb9f700088545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:52:59.773272 13823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d929df5ce8c14391af1eeb4c6c611f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:52:59.773106 13957 tablet_service.cc:1467] Processing CreateTablet for tablet d929df5ce8c14391af1eeb4c6c611f22 (DEFAULT_TABLE table=test-table [id=18596b7f911e458f9aedb9f700088545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:52:59.774220 13957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d929df5ce8c14391af1eeb4c6c611f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:52:59.774935 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:59.760441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:39055 (local address 127.0.0.1:45764)
0620 21:52:59.766679 (+ 6238us) negotiation.cc:107] Waiting for socket to connect
0620 21:52:59.766694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:52:59.766782 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:52:59.768778 (+ 1996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:52:59.768781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:52:59.768800 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:52:59.768891 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:59.768897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:59.771514 (+ 2617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:59.771518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:59.772826 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:52:59.772839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:52:59.774459 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:52:59.774462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:52:59.774561 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:59.774588 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:52:59.774671 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 21:52:59.774774 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6162,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260620 21:52:59.775461 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:52:59.760600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45764 (local address 127.0.193.193:39055)
0620 21:52:59.762324 (+ 1724us) server_negotiation.cc:206] Beginning negotiation
0620 21:52:59.762330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:52:59.766880 (+ 4550us) server_negotiation.cc:407] Connection header received
0620 21:52:59.766965 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:52:59.766970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:52:59.767217 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:52:59.767349 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:52:59.769026 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:59.771354 (+ 2328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:59.773004 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:52:59.773947 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:52:59.774076 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:52:59.774240 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 21:52:59.775089 (+ 849us) server_negotiation.cc:299] Negotiation successful
0620 21:52:59.775222 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1621,"thread_start_us":88,"threads_started":1}
I20260620 21:52:59.778985 13689 tablet_service.cc:1467] Processing CreateTablet for tablet d929df5ce8c14391af1eeb4c6c611f22 (DEFAULT_TABLE table=test-table [id=18596b7f911e458f9aedb9f700088545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:52:59.780189 13689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d929df5ce8c14391af1eeb4c6c611f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:52:59.785961 14056 tablet_bootstrap.cc:492] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02: Bootstrap starting.
I20260620 21:52:59.788575 14056 tablet_bootstrap.cc:654] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02: Neither blocks nor log segments found. Creating new log.
I20260620 21:52:59.790578 14056 log.cc:824] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02: Log is configured to *not* fsync() on all Append() calls
I20260620 21:52:59.791477 14057 tablet_bootstrap.cc:492] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18: Bootstrap starting.
I20260620 21:52:59.794121 14057 tablet_bootstrap.cc:654] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18: Neither blocks nor log segments found. Creating new log.
I20260620 21:52:59.794858 14058 tablet_bootstrap.cc:492] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844: Bootstrap starting.
I20260620 21:52:59.795066 14057 log.cc:824] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18: Log is configured to *not* fsync() on all Append() calls
I20260620 21:52:59.797214 14056 tablet_bootstrap.cc:492] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02: No bootstrap required, opened a new log
I20260620 21:52:59.797808 14058 tablet_bootstrap.cc:654] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844: Neither blocks nor log segments found. Creating new log.
I20260620 21:52:59.797467 14056 ts_tablet_manager.cc:1397] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260620 21:52:59.805438 14057 tablet_bootstrap.cc:492] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18: No bootstrap required, opened a new log
I20260620 21:52:59.805778 14057 ts_tablet_manager.cc:1397] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260620 21:52:59.803534 14056 raft_consensus.cc:348] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6559e16b69404159a9936d466c797b18" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40175 } } peers { permanent_uuid: "42ec5a99259c4d4691738cecba4fee02" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33095 } } peers { permanent_uuid: "755163dedcda4177ad0fdd4ddeb61844" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39055 } }
I20260620 21:52:59.809563 14056 raft_consensus.cc:374] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:52:59.809676 14056 raft_consensus.cc:729] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42ec5a99259c4d4691738cecba4fee02, State: Initialized, Role: FOLLOWER
I20260620 21:52:59.810298 14056 consensus_queue.cc:260] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6559e16b69404159a9936d466c797b18" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40175 } } peers { permanent_uuid: "42ec5a99259c4d4691738cecba4fee02" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33095 } } peers { permanent_uuid: "755163dedcda4177ad0fdd4ddeb61844" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39055 } }
I20260620 21:52:59.810788 14058 log.cc:824] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844: Log is configured to *not* fsync() on all Append() calls
I20260620 21:52:59.811626 14056 ts_tablet_manager.cc:1428] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260620 21:52:59.814117 14023 heartbeater.cc:503] Master 127.0.193.254:38539 was elected leader, sending a full tablet report...
I20260620 21:52:59.817611 14058 tablet_bootstrap.cc:492] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844: No bootstrap required, opened a new log
I20260620 21:52:59.817957 14058 ts_tablet_manager.cc:1397] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.002s
I20260620 21:52:59.823086 14057 raft_consensus.cc:348] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6559e16b69404159a9936d466c797b18" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40175 } } peers { permanent_uuid: "42ec5a99259c4d4691738cecba4fee02" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33095 } } peers { permanent_uuid: "755163dedcda4177ad0fdd4ddeb61844" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39055 } }
I20260620 21:52:59.823086 14058 raft_consensus.cc:348] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6559e16b69404159a9936d466c797b18" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40175 } } peers { permanent_uuid: "42ec5a99259c4d4691738cecba4fee02" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33095 } } peers { permanent_uuid: "755163dedcda4177ad0fdd4ddeb61844" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39055 } }
I20260620 21:52:59.823468 14057 raft_consensus.cc:374] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:52:59.823539 14057 raft_consensus.cc:729] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6559e16b69404159a9936d466c797b18, State: Initialized, Role: FOLLOWER
I20260620 21:52:59.823688 14058 raft_consensus.cc:374] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:52:59.823792 14058 raft_consensus.cc:729] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755163dedcda4177ad0fdd4ddeb61844, State: Initialized, Role: FOLLOWER
I20260620 21:52:59.824098 14057 consensus_queue.cc:260] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6559e16b69404159a9936d466c797b18" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40175 } } peers { permanent_uuid: "42ec5a99259c4d4691738cecba4fee02" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33095 } } peers { permanent_uuid: "755163dedcda4177ad0fdd4ddeb61844" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39055 } }
I20260620 21:52:59.824440 14058 consensus_queue.cc:260] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6559e16b69404159a9936d466c797b18" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40175 } } peers { permanent_uuid: "42ec5a99259c4d4691738cecba4fee02" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33095 } } peers { permanent_uuid: "755163dedcda4177ad0fdd4ddeb61844" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39055 } }
I20260620 21:52:59.825380 14057 ts_tablet_manager.cc:1428] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18: Time spent starting tablet: real 0.019s user 0.009s sys 0.000s
I20260620 21:52:59.825680 14058 ts_tablet_manager.cc:1428] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260620 21:52:59.825790 13889 heartbeater.cc:503] Master 127.0.193.254:38539 was elected leader, sending a full tablet report...
I20260620 21:52:59.826124 13755 heartbeater.cc:503] Master 127.0.193.254:38539 was elected leader, sending a full tablet report...
W20260620 21:52:59.887944 13890 tablet.cc:2369] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 21:52:59 dist-test-slave-kk5c krb5kdc[12561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for HTTP/127.0.193.212@KRBTEST.COM
W20260620 21:52:59.943572 14024 tablet.cc:2369] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:53:00.064095 13756 tablet.cc:2369] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:53:00.145330 14061 raft_consensus.cc:482] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:53:00.145578 14061 raft_consensus.cc:504] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6559e16b69404159a9936d466c797b18" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40175 } } peers { permanent_uuid: "42ec5a99259c4d4691738cecba4fee02" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33095 } } peers { permanent_uuid: "755163dedcda4177ad0fdd4ddeb61844" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39055 } }
I20260620 21:53:00.147055 14061 leader_election.cc:288] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6559e16b69404159a9936d466c797b18 (127.0.193.194:40175), 755163dedcda4177ad0fdd4ddeb61844 (127.0.193.193:39055)
I20260620 21:53:00.154039 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:00.147557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:40175 (local address 127.0.193.195:33465)
0620 21:53:00.149142 (+ 1585us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:00.149158 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:00.149325 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:00.149621 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:00.149624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:00.149647 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:00.149811 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:00.149819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:00.150999 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:00.151003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:00.152498 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:00.152509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:00.153555 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:00.153558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:00.153667 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:00.153701 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 21:53:00.153779 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 21:53:00.153854 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1513,"thread_start_us":136,"threads_started":1}
I20260620 21:53:00.154037 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:00.148694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:33465 (local address 127.0.193.194:40175)
0620 21:53:00.148831 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:00.148836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:00.149256 (+ 420us) server_negotiation.cc:407] Connection header received
0620 21:53:00.149428 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:00.149431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:00.149486 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:00.149586 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:00.150029 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:00.150863 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:00.152651 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:00.153294 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:00.153401 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:00.153454 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:00.153824 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 21:53:00.153908 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:53:00.155174 13843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d929df5ce8c14391af1eeb4c6c611f22" candidate_uuid: "42ec5a99259c4d4691738cecba4fee02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6559e16b69404159a9936d466c797b18" is_pre_election: true
I20260620 21:53:00.155545 13843 raft_consensus.cc:2454] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42ec5a99259c4d4691738cecba4fee02 in term 0.
I20260620 21:53:00.155786 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:00.148715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:35223 (local address 127.0.193.193:39055)
0620 21:53:00.148955 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:00.148960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:00.151544 (+ 2584us) server_negotiation.cc:407] Connection header received
0620 21:53:00.151727 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:00.151730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:00.151781 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:00.151831 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:00.152250 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:00.153038 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:00.154434 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:00.155046 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:00.155233 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:00.155283 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:00.155586 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 21:53:00.155664 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260620 21:53:00.156149 13912 leader_election.cc:302] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42ec5a99259c4d4691738cecba4fee02, 6559e16b69404159a9936d466c797b18; no voters:
I20260620 21:53:00.156270 14070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:00.148040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:39055 (local address 127.0.193.195:35223)
0620 21:53:00.151447 (+ 3407us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:00.151464 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:00.151587 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:00.151970 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:00.151973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:00.151992 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:00.152110 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:00.152117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:00.153191 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:00.153194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:00.154311 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:00.154320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:00.155344 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:00.155347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:00.155450 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:00.155472 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:53:00.156006 (+ 534us) client_negotiation.cc:240] Negotiation successful
0620 21:53:00.156092 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3324,"thread_start_us":96,"threads_started":1}
I20260620 21:53:00.156430 14061 raft_consensus.cc:2781] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:53:00.156522 14061 raft_consensus.cc:482] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:53:00.156562 14061 raft_consensus.cc:3037] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:00.156940 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d929df5ce8c14391af1eeb4c6c611f22" candidate_uuid: "42ec5a99259c4d4691738cecba4fee02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755163dedcda4177ad0fdd4ddeb61844" is_pre_election: true
I20260620 21:53:00.157213 13709 raft_consensus.cc:2454] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42ec5a99259c4d4691738cecba4fee02 in term 0.
I20260620 21:53:00.157873 14061 raft_consensus.cc:504] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6559e16b69404159a9936d466c797b18" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40175 } } peers { permanent_uuid: "42ec5a99259c4d4691738cecba4fee02" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33095 } } peers { permanent_uuid: "755163dedcda4177ad0fdd4ddeb61844" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39055 } }
I20260620 21:53:00.158257 14061 leader_election.cc:288] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [CANDIDATE]: Term 1 election: Requested vote from peers 6559e16b69404159a9936d466c797b18 (127.0.193.194:40175), 755163dedcda4177ad0fdd4ddeb61844 (127.0.193.193:39055)
I20260620 21:53:00.158651 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d929df5ce8c14391af1eeb4c6c611f22" candidate_uuid: "42ec5a99259c4d4691738cecba4fee02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755163dedcda4177ad0fdd4ddeb61844"
I20260620 21:53:00.158639 13843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d929df5ce8c14391af1eeb4c6c611f22" candidate_uuid: "42ec5a99259c4d4691738cecba4fee02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6559e16b69404159a9936d466c797b18"
I20260620 21:53:00.158767 13709 raft_consensus.cc:3037] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:00.158771 13843 raft_consensus.cc:3037] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:00.160017 13709 raft_consensus.cc:2454] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42ec5a99259c4d4691738cecba4fee02 in term 1.
I20260620 21:53:00.160153 13843 raft_consensus.cc:2454] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42ec5a99259c4d4691738cecba4fee02 in term 1.
I20260620 21:53:00.160360 13911 leader_election.cc:302] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42ec5a99259c4d4691738cecba4fee02, 755163dedcda4177ad0fdd4ddeb61844; no voters:
I20260620 21:53:00.160532 14061 raft_consensus.cc:2781] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:53:00.160732 14061 raft_consensus.cc:686] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [term 1 LEADER]: Becoming Leader. State: Replica: 42ec5a99259c4d4691738cecba4fee02, State: Running, Role: LEADER
I20260620 21:53:00.161084 14061 consensus_queue.cc:237] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6559e16b69404159a9936d466c797b18" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40175 } } peers { permanent_uuid: "42ec5a99259c4d4691738cecba4fee02" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33095 } } peers { permanent_uuid: "755163dedcda4177ad0fdd4ddeb61844" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39055 } }
I20260620 21:53:00.164371 13556 catalog_manager.cc:5495] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42ec5a99259c4d4691738cecba4fee02 (127.0.193.195). New cstate: current_term: 1 leader_uuid: "42ec5a99259c4d4691738cecba4fee02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6559e16b69404159a9936d466c797b18" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42ec5a99259c4d4691738cecba4fee02" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 33095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755163dedcda4177ad0fdd4ddeb61844" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39055 } health_report { overall_health: UNKNOWN } } }
I20260620 21:53:00.218407 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:00.214830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.0.193.195:33095)
0620 21:53:00.214971 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:00.214974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:00.214988 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:53:00.215133 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:00.215135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:00.215260 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:00.215350 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:53:00.215604 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:00.216114 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:00.216999 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:00.217208 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:00.217285 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:00.217585 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:53:00.218008 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:53:00.218098 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:00.218215 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 21:53:00.218267 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:53:00.224287 13843 raft_consensus.cc:1261] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18 [term 1 FOLLOWER]: Refusing update from remote peer 42ec5a99259c4d4691738cecba4fee02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:53:00.224287 13709 raft_consensus.cc:1261] T d929df5ce8c14391af1eeb4c6c611f22 P 755163dedcda4177ad0fdd4ddeb61844 [term 1 FOLLOWER]: Refusing update from remote peer 42ec5a99259c4d4691738cecba4fee02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:53:00.224992 14073 consensus_queue.cc:1035] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755163dedcda4177ad0fdd4ddeb61844" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:53:00.225183 14061 consensus_queue.cc:1035] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6559e16b69404159a9936d466c797b18" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:53:00.235697 14080 mvcc.cc:205] Tried to move back new op lower bound from 7299040789391253504 to 7299040789142708224. Current Snapshot: MvccSnapshot[applied={T|T < 7299040789391253504}]
I20260620 21:53:00.236356 14081 mvcc.cc:205] Tried to move back new op lower bound from 7299040789391253504 to 7299040789142708224. Current Snapshot: MvccSnapshot[applied={T|T < 7299040789391253504}]
I20260620 21:53:00.241696 13556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:53:00.241880 13556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:53:00.244270 13556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b21c72b40078451f86a4db7eb1de6120: Sending DeleteTablet for 3 replicas of tablet d929df5ce8c14391af1eeb4c6c611f22
I20260620 21:53:00.245113 13823 tablet_service.cc:1514] Processing DeleteTablet for tablet d929df5ce8c14391af1eeb4c6c611f22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:00 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:54628
I20260620 21:53:00.245218 13689 tablet_service.cc:1514] Processing DeleteTablet for tablet d929df5ce8c14391af1eeb4c6c611f22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:00 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:45764
I20260620 21:53:00.245702 14088 tablet_replica.cc:330] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18: stopping tablet replica
I20260620 21:53:00.245679 13957 tablet_service.cc:1514] Processing DeleteTablet for tablet d929df5ce8c14391af1eeb4c6c611f22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:00 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:35826
I20260620 21:53:00.245782 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 13595
I20260620 21:53:00.246361 14090 tablet_replica.cc:330] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02: stopping tablet replica
I20260620 21:53:00.246718 14090 raft_consensus.cc:2229] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:53:00.247233 14090 raft_consensus.cc:2258] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:53:00.249059 14088 raft_consensus.cc:2229] T d929df5ce8c14391af1eeb4c6c611f22 P 6559e16b69404159a9936d466c797b18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:53:00.250211 14090 ts_tablet_manager.cc:1905] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 21:53:00.253698 13542 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:39055 (error 108)
I20260620 21:53:00.253772 14090 ts_tablet_manager.cc:1918] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:53:00.253870 14090 log.cc:1196] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/d929df5ce8c14391af1eeb4c6c611f22
I20260620 21:53:00.253916 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 13759
I20260620 21:53:00.255415 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:00.254781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:39055 (local address 127.0.0.1:45770)
0620 21:53:00.255212 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:00.255300 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:39055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278}
W20260620 21:53:00.256132 13542 catalog_manager.cc:4584] TS 755163dedcda4177ad0fdd4ddeb61844 (127.0.193.193:39055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d929df5ce8c14391af1eeb4c6c611f22: Network error: Client connection negotiation failed: client connection to 127.0.193.193:39055: connect: Connection refused (error 111)
I20260620 21:53:00.259565 14090 ts_tablet_manager.cc:1939] T d929df5ce8c14391af1eeb4c6c611f22 P 42ec5a99259c4d4691738cecba4fee02: Deleting consensus metadata
I20260620 21:53:00.260793 13541 catalog_manager.cc:4841] TS 42ec5a99259c4d4691738cecba4fee02 (127.0.193.195:33095): tablet d929df5ce8c14391af1eeb4c6c611f22 (table test-table [id=18596b7f911e458f9aedb9f700088545]) successfully deleted
I20260620 21:53:00.261955 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 13891
I20260620 21:53:00.263907 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:00.262148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:40175 (local address 127.0.0.1:54632)
0620 21:53:00.263772 (+ 1624us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:00.263830 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.194:40175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1539}
I20260620 21:53:00.268188 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 13524
I20260620 21:53:00.277644 775 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 21:53:00.613451 775 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 21:53:00.613621 775 mini_ranger.cc:66] Stopping Ranger...
I20260620 21:53:00.959471 775 mini_ranger.cc:68] Stopped Ranger
2026-06-20 21:53:00.959 UTC [12591] LOG: received smart shutdown request
2026-06-20 21:53:00.964 UTC [12591] LOG: background worker "logical replication launcher" (PID 12600) exited with exit code 1
2026-06-20 21:53:00.964 UTC [12595] LOG: shutting down
2026-06-20 21:53:01.167 UTC [12591] LOG: database system is shut down
2026-06-20T21:53:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54101 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:53:01 dist-test-slave-kk5c krb5kdc[14100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:53:01 dist-test-slave-kk5c krb5kdc[14100](info): set up 2 sockets
Jun 20 21:53:01 dist-test-slave-kk5c krb5kdc[14100](info): commencing operation
krb5kdc: starting...
W20260620 21:53:02.391433 775 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:53:02 dist-test-slave-kk5c krb5kdc[14100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:53:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:53:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:53:02.523386 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:39275
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:39275,127.0.193.253:40905,127.0.193.252:35073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:53:02.639540 14113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:02.639824 14113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:02.639909 14113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:02.644052 14113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:53:02.644119 14113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:02.644138 14113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:02.644156 14113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:53:02.644172 14113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:53:02.650269 14113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:39275,127.0.193.253:40905,127.0.193.252:35073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:39275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:02.651511 14113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:02.652604 14113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:53:02.661796 14113 server_base.cc:1034] running on GCE node
W20260620 21:53:02.661765 14118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:02.661832 14121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:02.661991 14119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:02.662876 14113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:02.664216 14113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:02.665424 14113 hybrid_clock.cc:648] HybridClock initialized: now 1781992382665429 us; error 52 us; skew 500 ppm
Jun 20 21:53:02 dist-test-slave-kk5c krb5kdc[14100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:02.668639 14113 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:53:02.669627 14113 webserver.cc:460] Webserver started at http://127.0.193.254:38389/ using document root <none> and password file <none>
I20260620 21:53:02.670204 14113 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:02.670269 14113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:02.670449 14113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:53:02.672789 14113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "6561c7a3110544268c23ea7da21bbb0d"
format_stamp: "Formatted at 2026-06-20 21:53:02 on dist-test-slave-kk5c"
server_key: "639af2cd62e83d423b1368b59ef4ac57"
server_key_iv: "69ee30d20750fa81ac7ecc8fe44ae0ce"
server_key_version: "encryptionkey@0"
I20260620 21:53:02.673367 14113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "6561c7a3110544268c23ea7da21bbb0d"
format_stamp: "Formatted at 2026-06-20 21:53:02 on dist-test-slave-kk5c"
server_key: "639af2cd62e83d423b1368b59ef4ac57"
server_key_iv: "69ee30d20750fa81ac7ecc8fe44ae0ce"
server_key_version: "encryptionkey@0"
I20260620 21:53:02.677711 14113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 21:53:02.680615 14128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:02.682083 14113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 21:53:02.682235 14113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "6561c7a3110544268c23ea7da21bbb0d"
format_stamp: "Formatted at 2026-06-20 21:53:02 on dist-test-slave-kk5c"
server_key: "639af2cd62e83d423b1368b59ef4ac57"
server_key_iv: "69ee30d20750fa81ac7ecc8fe44ae0ce"
server_key_version: "encryptionkey@0"
I20260620 21:53:02.682343 14113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:02.705718 14113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:02.709259 14113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:02.709441 14113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:02.718008 14113 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:39275
I20260620 21:53:02.718041 14180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:39275 every 8 connection(s)
I20260620 21:53:02.719115 14113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:53:02.720618 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 14113
I20260620 21:53:02.720795 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:53:02.721174 775 external_mini_cluster.cc:1428] Setting key 49b0d8e748c217681139429fb4de867d
I20260620 21:53:02.722431 14181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:02.726871 14181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.193.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.253@KRBTEST.COM" created.
Jun 20 21:53:02 dist-test-slave-kk5c krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:02.743314 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.730256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.0.193.254:39275)
0620 21:53:02.730644 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:02.730662 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:02.730724 (+ 62us) server_negotiation.cc:407] Connection header received
0620 21:53:02.731305 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:02.731310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:02.731516 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:02.731754 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:02.732776 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.733681 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.734718 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.735107 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.735300 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:02.737738 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:02.737744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:02.737755 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:02.737776 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:02.739846 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:02.740380 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:02.740393 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:02.740395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:02.740497 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:02.740820 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:02.740826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:02.740827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:02.741246 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:02.741389 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:02.741774 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 21:53:02.742124 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":210,"thread_start_us":97,"threads_started":1}
I20260620 21:53:02.743311 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.729254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39275 (local address 127.0.0.1:57892)
0620 21:53:02.729810 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:02.729868 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:02.730958 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:02.731765 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:02.731770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:02.732239 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:02.732580 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:02.732617 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.733827 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.733831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:02.734594 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:02.734603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.735289 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.735293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:02.735383 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:02.736034 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:02.736053 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:02.737569 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:02.739985 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:02.739993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:02.740013 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:02.740259 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:02.740606 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:02.740609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:02.740611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:02.740725 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:02.741369 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:02.741375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:02.741521 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 21:53:02.741700 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 21:53:02.742171 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":158,"thread_start_us":173,"threads_started":1}
I20260620 21:53:02.746594 14181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:53:02.747556 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.747233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40905 (local address 127.0.0.1:33300)
0620 21:53:02.747400 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:02.747481 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 21:53:02.747823 14132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40905: connect: Connection refused (error 111)
I20260620 21:53:02.748905 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.748688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40905 (local address 127.0.0.1:33316)
0620 21:53:02.748795 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:02.748843 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 21:53:02.749270 14181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.253:40905: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40905: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.253.keytab.
Entry for principal kudu/127.0.193.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.253.keytab.
I20260620 21:53:02.752275 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.253:40905
--webserver_interface=127.0.193.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.253.keytab
--principal=kudu/127.0.193.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:39275,127.0.193.253:40905,127.0.193.252:35073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:53:02.798602 14181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } attempt: 1
I20260620 21:53:02.799727 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.799342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40905 (local address 127.0.0.1:33328)
0620 21:53:02.799583 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:02.799647 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260620 21:53:02.800597 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.800372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40905 (local address 127.0.0.1:33342)
0620 21:53:02.800489 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:02.800542 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 21:53:02.800712 14181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.253:40905: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40905: connect: Connection refused (error 111)
I20260620 21:53:02.868894 14181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } attempt: 2
I20260620 21:53:02.869791 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.869442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40905 (local address 127.0.0.1:33348)
0620 21:53:02.869623 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:02.869698 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 21:53:02.870707 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.870462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40905 (local address 127.0.0.1:33354)
0620 21:53:02.870584 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:02.870639 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 21:53:02.870863 14181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.253:40905: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40905: connect: Connection refused (error 111)
W20260620 21:53:02.873188 14188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:02.873449 14188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:02.873544 14188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:02.877969 14188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:53:02.878050 14188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:02.878072 14188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:02.878088 14188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:53:02.878136 14188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:53:02.884088 14188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:39275,127.0.193.253:40905,127.0.193.252:35073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.253.keytab
--principal=kudu/127.0.193.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.253:40905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.193.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:02.885259 14188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:02.886407 14188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:02.895797 14193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:02.895797 14194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:02.895835 14196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:02.896157 14188 server_base.cc:1034] running on GCE node
I20260620 21:53:02.896853 14188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:02.898059 14188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:02.899266 14188 hybrid_clock.cc:648] HybridClock initialized: now 1781992382899251 us; error 29 us; skew 500 ppm
Jun 20 21:53:02 dist-test-slave-kk5c krb5kdc[14100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:02.902403 14188 init.cc:377] Logged in from keytab as kudu/127.0.193.253@KRBTEST.COM (short username kudu)
I20260620 21:53:02.903546 14188 webserver.cc:460] Webserver started at http://127.0.193.253:42745/ using document root <none> and password file <none>
I20260620 21:53:02.904136 14188 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:02.904201 14188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:02.904417 14188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:53:02.906514 14188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/data/instance:
uuid: "251aa2d9b54a468d82abe1ac00918162"
format_stamp: "Formatted at 2026-06-20 21:53:02 on dist-test-slave-kk5c"
server_key: "75aceb5ecfd414e5e97ca53a2ad8d8dc"
server_key_iv: "e8ff37ed486f1f5c5090e63238d8ea4e"
server_key_version: "encryptionkey@0"
I20260620 21:53:02.907107 14188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/wal/instance:
uuid: "251aa2d9b54a468d82abe1ac00918162"
format_stamp: "Formatted at 2026-06-20 21:53:02 on dist-test-slave-kk5c"
server_key: "75aceb5ecfd414e5e97ca53a2ad8d8dc"
server_key_iv: "e8ff37ed486f1f5c5090e63238d8ea4e"
server_key_version: "encryptionkey@0"
I20260620 21:53:02.911108 14188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:53:02.913955 14203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:02.915514 14188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 21:53:02.915683 14188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/wal
uuid: "251aa2d9b54a468d82abe1ac00918162"
format_stamp: "Formatted at 2026-06-20 21:53:02 on dist-test-slave-kk5c"
server_key: "75aceb5ecfd414e5e97ca53a2ad8d8dc"
server_key_iv: "e8ff37ed486f1f5c5090e63238d8ea4e"
server_key_version: "encryptionkey@0"
I20260620 21:53:02.915818 14188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:02.942508 14188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:02.946389 14188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:02.946631 14188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:02.955062 14188 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.253:40905
I20260620 21:53:02.955088 14255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.253:40905 every 8 connection(s)
I20260620 21:53:02.956331 14188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/data/info.pb
I20260620 21:53:02.959306 14256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:02.959903 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 14188
I20260620 21:53:02.960049 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-1/wal/instance
I20260620 21:53:02.960417 775 external_mini_cluster.cc:1428] Setting key 5f86c174e5fe3ecfc3568f1000f2f2f6
I20260620 21:53:02.962159 14181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } attempt: 3
I20260620 21:53:02.964478 14256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.193.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.252@KRBTEST.COM" created.
Jun 20 21:53:02 dist-test-slave-kk5c krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for kudu/127.0.193.253@KRBTEST.COM
Jun 20 21:53:02 dist-test-slave-kk5c krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.253@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:02.978536 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.962861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40905 (local address 127.0.0.1:33366)
0620 21:53:02.963117 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:02.963144 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:02.964549 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:02.966206 (+ 1657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:02.966214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:02.966469 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:02.966682 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:02.966695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.968664 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.968668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:02.969203 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:02.969213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.970393 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.970397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:02.970553 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:02.971252 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:02.971276 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:02.973930 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:02.976361 (+ 2431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:02.976365 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:02.976367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:02.976726 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:02.977088 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:02.977091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:02.977093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:02.977183 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:02.977965 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:02.977968 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:02.978098 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 21:53:02.978213 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 21:53:02.978363 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":38}
I20260620 21:53:02.979076 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.966978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.0.193.254:39275)
0620 21:53:02.967115 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:02.967120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:02.967134 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:53:02.967176 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:02.967179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:02.967263 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:02.967365 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:02.968259 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.969188 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.970316 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.971110 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.971257 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:02.974695 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:02.974701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:02.974704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:02.974737 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:02.976759 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:02.977347 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:02.977350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:02.977351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:02.977419 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:02.977791 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:02.977797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:02.977800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:02.978081 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:02.978162 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:02.978681 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 21:53:02.978832 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":57}
I20260620 21:53:02.979796 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.963966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.0.193.253:40905)
0620 21:53:02.964528 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:02.964559 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:02.964606 (+ 47us) server_negotiation.cc:407] Connection header received
0620 21:53:02.965430 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:02.965452 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:02.965872 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:02.966217 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:02.967225 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.968551 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.969378 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.970230 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.970427 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:02.974133 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:02.974144 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:02.974163 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:02.974199 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:02.976233 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:02.976849 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:02.976857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:02.976868 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:02.976976 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:02.977272 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:02.977277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:02.977279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:02.977859 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:02.978001 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:02.978372 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 21:53:02.978723 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":388,"thread_start_us":179,"threads_started":1}
I20260620 21:53:02.981076 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.966499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39275 (local address 127.0.0.1:57896)
0620 21:53:02.966914 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:02.966964 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:02.967068 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:02.967432 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:02.967440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:02.967886 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:02.968089 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:02.968102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.969340 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.969346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:02.970178 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:02.970188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.971301 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.971305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:02.971412 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:02.972026 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:02.972045 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:02.974487 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:02.976897 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:02.976909 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:02.976919 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:02.977248 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:02.977539 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:02.977545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:02.977550 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:02.977663 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:02.978254 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:02.978260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:02.978405 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 21:53:02.979529 (+ 1124us) client_negotiation.cc:240] Negotiation successful
0620 21:53:02.979778 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":60,"thread_start_us":130,"threads_started":1}
I20260620 21:53:02.982287 14181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:53:02.982697 14256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:53:02.983570 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.983166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33716)
0620 21:53:02.983368 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:02.983483 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260620 21:53:02.984504 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.984256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33728)
0620 21:53:02.984406 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:02.984449 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260620 21:53:02.984668 14181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:35073: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
Jun 20 21:53:02 dist-test-slave-kk5c krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.253@KRBTEST.COM for kudu/127.0.193.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.252.keytab.
Entry for principal kudu/127.0.193.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.252.keytab.
I20260620 21:53:02.990589 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.252:35073
--webserver_interface=127.0.193.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.252.keytab
--principal=kudu/127.0.193.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:39275,127.0.193.253:40905,127.0.193.252:35073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:53:02.993827 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.983342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33372 (local address 127.0.193.253:40905)
0620 21:53:02.983448 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:02.983452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:02.983466 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:53:02.983569 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:02.983572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:02.983627 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:02.983704 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:02.984259 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.985253 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.986017 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.986343 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.986469 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:02.988707 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:02.988710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:02.988712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:02.988738 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:02.992070 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:02.992551 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:02.992556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:02.992558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:02.992643 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:02.992920 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:02.992923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:02.992924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:02.993119 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:02.993225 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:02.993494 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 21:53:02.993696 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":43}
I20260620 21:53:02.993827 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.983175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40905 (local address 127.0.0.1:33372)
0620 21:53:02.983325 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:02.983341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:02.983467 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:02.983809 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:02.983812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:02.984048 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:02.984143 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:02.984149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.985377 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.985379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:02.985917 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:02.985925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:02.986488 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:02.986492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:02.986608 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:02.987178 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:02.987233 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:02.988455 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:02.992205 (+ 3750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:02.992222 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:02.992229 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:02.992439 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:02.992741 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:02.992745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:02.992752 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:02.992822 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:02.993233 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:02.993239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:02.993323 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 21:53:02.993450 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 21:53:02.993629 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":1}
I20260620 21:53:02.994932 14256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:53:02.995844 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.995482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33732)
0620 21:53:02.995628 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:02.995740 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 21:53:02.996197 14204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
I20260620 21:53:02.997187 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:02.996921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33736)
0620 21:53:02.997072 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:02.997120 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 21:53:02.997437 14256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:35073: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
I20260620 21:53:03.015841 14181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } attempt: 1
I20260620 21:53:03.016755 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.016393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33738)
0620 21:53:03.016587 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:03.016658 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 21:53:03.017584 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.017336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33744)
0620 21:53:03.017459 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:03.017521 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 21:53:03.017724 14181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:35073: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
I20260620 21:53:03.046789 14256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } attempt: 1
I20260620 21:53:03.047788 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.047497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33750)
0620 21:53:03.047648 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:03.047714 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 21:53:03.048658 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.048406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33766)
0620 21:53:03.048545 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:03.048600 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 21:53:03.048844 14256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:35073: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
I20260620 21:53:03.092909 14181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } attempt: 2
I20260620 21:53:03.093827 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.093480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33768)
0620 21:53:03.093657 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:03.093731 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 21:53:03.094583 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.094357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33770)
0620 21:53:03.094465 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:03.094518 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 21:53:03.094717 14181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:35073: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
W20260620 21:53:03.109264 14262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:03.109579 14262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:03.109622 14262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:03.114068 14262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:53:03.114166 14262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:03.114187 14262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:03.114204 14262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:53:03.114220 14262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:53:03.117028 14256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } attempt: 2
I20260620 21:53:03.117880 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.117587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33772)
0620 21:53:03.117739 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:03.117807 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 21:53:03.118603 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.118383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33784)
0620 21:53:03.118491 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:03.118544 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 21:53:03.118737 14256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:35073: Network error: Client connection negotiation failed: client connection to 127.0.193.252:35073: connect: Connection refused (error 111)
I20260620 21:53:03.120409 14262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:39275,127.0.193.253:40905,127.0.193.252:35073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.252.keytab
--principal=kudu/127.0.193.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.252:35073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.193.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:03.121587 14262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:03.122681 14262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:03.131611 14269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:03.131621 14268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:03.131621 14271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:03.131879 14262 server_base.cc:1034] running on GCE node
I20260620 21:53:03.132643 14262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:03.134084 14262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:03.135293 14262 hybrid_clock.cc:648] HybridClock initialized: now 1781992383135263 us; error 39 us; skew 500 ppm
Jun 20 21:53:03 dist-test-slave-kk5c krb5kdc[14100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:03.138420 14262 init.cc:377] Logged in from keytab as kudu/127.0.193.252@KRBTEST.COM (short username kudu)
I20260620 21:53:03.139595 14262 webserver.cc:460] Webserver started at http://127.0.193.252:43941/ using document root <none> and password file <none>
I20260620 21:53:03.140229 14262 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:03.140332 14262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:03.140594 14262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:53:03.142916 14262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/data/instance:
uuid: "56ab9aaca87d42e0bab94e63269c2a9a"
format_stamp: "Formatted at 2026-06-20 21:53:03 on dist-test-slave-kk5c"
server_key: "b055f0ca26e9d98a250beffada0dc761"
server_key_iv: "27e1ce95688477222ce02b94cd9ffbf9"
server_key_version: "encryptionkey@0"
I20260620 21:53:03.143625 14262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/wal/instance:
uuid: "56ab9aaca87d42e0bab94e63269c2a9a"
format_stamp: "Formatted at 2026-06-20 21:53:03 on dist-test-slave-kk5c"
server_key: "b055f0ca26e9d98a250beffada0dc761"
server_key_iv: "27e1ce95688477222ce02b94cd9ffbf9"
server_key_version: "encryptionkey@0"
I20260620 21:53:03.148576 14262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 21:53:03.151743 14278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:03.153246 14262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:53:03.153406 14262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/wal
uuid: "56ab9aaca87d42e0bab94e63269c2a9a"
format_stamp: "Formatted at 2026-06-20 21:53:03 on dist-test-slave-kk5c"
server_key: "b055f0ca26e9d98a250beffada0dc761"
server_key_iv: "27e1ce95688477222ce02b94cd9ffbf9"
server_key_version: "encryptionkey@0"
I20260620 21:53:03.153558 14262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:03.176453 14262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:03.180821 14262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:03.181099 14262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:03.189953 14262 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.252:35073
I20260620 21:53:03.189968 14330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.252:35073 every 8 connection(s)
I20260620 21:53:03.191236 14262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/data/info.pb
I20260620 21:53:03.193907 14181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } attempt: 3
I20260620 21:53:03.194337 14331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:03.198189 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 14262
I20260620 21:53:03.198330 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781992210835692-775-0/minicluster-data/master-2/wal/instance
I20260620 21:53:03.198593 775 external_mini_cluster.cc:1428] Setting key 9a7fdae00cc3f3a00f21c5d0f027ed4b
I20260620 21:53:03.200309 14331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:53:03 dist-test-slave-kk5c krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for kudu/127.0.193.252@KRBTEST.COM
Jun 20 21:53:03 dist-test-slave-kk5c krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.252@KRBTEST.COM
I20260620 21:53:03.209226 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.194525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33800)
0620 21:53:03.194692 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:03.194711 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:03.194828 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:03.197668 (+ 2840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:03.197673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:03.197918 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:03.198063 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:03.198073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.200203 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.200207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:03.201017 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:03.201031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.201983 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.201988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:03.202078 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.202654 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:03.202671 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:03.204385 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:03.207029 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:03.207033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:03.207035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:03.207353 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:03.207799 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:03.207805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:03.207807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:03.207912 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:03.208677 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:03.208680 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:03.208777 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 21:53:03.208893 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 21:53:03.209050 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
Jun 20 21:53:03 dist-test-slave-kk5c krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.252@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:03.209897 14256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } attempt: 3
I20260620 21:53:03.210412 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.195616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.0.193.252:35073)
0620 21:53:03.196166 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:03.196181 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:03.196221 (+ 40us) server_negotiation.cc:407] Connection header received
0620 21:53:03.196929 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:03.196946 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:03.197291 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:03.197601 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:03.198265 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.200028 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.201172 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.201840 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.202000 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.204577 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:03.204586 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:03.204601 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:03.204643 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:03.206893 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.207507 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.207516 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.207523 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.207674 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.208020 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.208026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.208029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.208490 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:03.208626 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:03.209071 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 21:53:03.209386 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":171,"threads_started":1}
I20260620 21:53:03.214274 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.202733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57898 (local address 127.0.193.254:39275)
0620 21:53:03.203159 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:03.203166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:03.203181 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:53:03.203248 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:03.203250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:03.203286 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:03.203388 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:03.204279 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.204837 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.206504 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.206803 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.206930 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.210179 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:03.210183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:03.210185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:03.210209 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:03.211888 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.212819 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.212822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.212825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.212920 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.213233 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.213236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.213238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.213526 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:03.213641 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:03.213989 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 21:53:03.214113 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339}
I20260620 21:53:03.214289 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.202378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39275 (local address 127.0.0.1:57898)
0620 21:53:03.202940 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:03.202991 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:03.203112 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:03.203462 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:03.203468 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:03.203940 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:03.204139 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:03.204149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.205284 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.205289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:03.206324 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:03.206334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.206928 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.206932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:03.207316 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.207796 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:03.207808 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:03.209965 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:03.212434 (+ 2469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:03.212440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:03.212445 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:03.212721 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:03.213025 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:03.213028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:03.213030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:03.213124 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:03.213650 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:03.213660 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:03.213772 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 21:53:03.213930 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 21:53:03.214114 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":98,"threads_started":1}
I20260620 21:53:03.215423 14181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d: Bootstrap starting.
I20260620 21:53:03.215873 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.199289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.0.193.252:35073)
0620 21:53:03.199664 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:03.199674 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:03.199789 (+ 115us) server_negotiation.cc:407] Connection header received
0620 21:53:03.199964 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:03.199967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:03.200013 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:03.200109 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:03.200871 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.201404 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.202411 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.202967 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.203068 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.206126 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:03.206130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:03.206132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:03.206162 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:03.207937 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.208471 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.208475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.208476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.208579 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.211690 (+ 3111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.211695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.211700 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.211913 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:03.212099 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:03.212492 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 21:53:03.212604 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"spinlock_wait_cycles":4352,"thread_start_us":141,"threads_started":1}
Jun 20 21:53:03 dist-test-slave-kk5c krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.253@KRBTEST.COM for kudu/127.0.193.252@KRBTEST.COM
I20260620 21:53:03.218287 14181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:03.219389 14331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:53:03.219816 14181 log.cc:824] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:03.221429 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.210334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33826)
0620 21:53:03.210613 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:03.210630 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:03.210721 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:03.211653 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:03.211657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:03.211856 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:03.211996 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:03.212004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.213513 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.213517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:03.214327 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:03.214342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.215563 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.215566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:03.215639 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.216160 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:03.216176 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:03.217675 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:03.219698 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:03.219701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:03.219704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:03.219987 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:03.220332 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:03.220337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:03.220339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:03.220450 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:03.220922 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:03.220924 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:03.221015 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 21:53:03.221109 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 21:53:03.221256 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260620 21:53:03.222057 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.210856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.0.193.252:35073)
0620 21:53:03.211424 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:03.211429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:03.211443 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:53:03.211477 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:03.211480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:03.211522 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:03.211604 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:03.212124 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.213383 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.214675 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.215166 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.215298 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.217785 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:03.217789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:03.217792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:03.217814 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:03.219498 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.220113 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.220117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.220119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.220205 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.220548 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.220554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.220557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.220809 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:03.220896 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:03.221736 (+ 840us) server_negotiation.cc:299] Negotiation successful
0620 21:53:03.221877 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":131,"threads_started":1}
I20260620 21:53:03.225369 14256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162: Bootstrap starting.
I20260620 21:53:03.225898 14181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d: No bootstrap required, opened a new log
I20260620 21:53:03.227788 14256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162: Neither blocks nor log segments found. Creating new log.
Jun 20 21:53:03 dist-test-slave-kk5c krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:03.228557 14256 log.cc:824] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:53:03 dist-test-slave-kk5c krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.252@KRBTEST.COM for kudu/127.0.193.253@KRBTEST.COM
I20260620 21:53:03.229861 14181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } }
I20260620 21:53:03.231035 14181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:03.231114 14181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6561c7a3110544268c23ea7da21bbb0d, State: Initialized, Role: FOLLOWER
I20260620 21:53:03.231463 14256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162: No bootstrap required, opened a new log
I20260620 21:53:03.231668 14181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } }
I20260620 21:53:03.233179 14181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:53:03.233925 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.219979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40905 (local address 127.0.0.1:33382)
0620 21:53:03.220410 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:03.220432 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:03.220510 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:03.220727 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:03.220730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:03.220916 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:03.221021 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:03.221026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.222028 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.222032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:03.222693 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:03.222702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.223373 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.223379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:03.223491 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.223950 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:03.223966 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:03.229521 (+ 5555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:03.231775 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:03.231779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:03.231783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:03.232070 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:03.232512 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:03.232516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:03.232518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:03.232611 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:03.233397 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:03.233400 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:03.233508 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 21:53:03.233593 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 21:53:03.233745 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365}
I20260620 21:53:03.234176 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.217439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.0.193.254:39275)
0620 21:53:03.217550 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:03.217554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:03.217568 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:53:03.217684 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:03.217687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:03.217730 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:03.217806 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:03.219328 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.220027 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.221318 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.221643 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.221725 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.229867 (+ 8142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:03.229873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:03.229880 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:03.229926 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:03.231670 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.232263 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.232266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.232268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.232362 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.232675 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.232678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.232679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.232925 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:03.233100 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:03.233825 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 21:53:03.233970 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:53:03.234973 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.220268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.0.193.253:40905)
0620 21:53:03.220402 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:03.220406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:03.220472 (+ 66us) server_negotiation.cc:407] Connection header received
0620 21:53:03.220595 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:03.220598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:03.220641 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:03.220712 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:03.221135 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.221901 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.222826 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.223243 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.223362 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.229698 (+ 6336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:03.229702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:03.229707 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:03.229739 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:03.231524 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.232310 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.232313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.232315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.232398 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.232716 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.232719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.232720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.233226 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:03.234596 (+ 1370us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:03.234674 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 21:53:03.234798 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 21:53:03.235805 14331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:53:03.236035 14256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } }
I20260620 21:53:03.236268 14256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:03.236333 14256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 251aa2d9b54a468d82abe1ac00918162, State: Initialized, Role: FOLLOWER
I20260620 21:53:03.236902 14256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } }
W20260620 21:53:03.240042 14351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
Jun 20 21:53:03 dist-test-slave-kk5c krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.252@KRBTEST.COM for kudu/127.0.193.252@KRBTEST.COM
W20260620 21:53:03.242678 14351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:53:03.240350 14340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } } }
I20260620 21:53:03.243042 14340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:53:03.238517 14113 master_runner.cc:186] Error getting master registration for 127.0.193.253:40905: OK, instance_id { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" instance_seqno: 1781992382952427 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:53:03.243875 14113 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 21:53:03 dist-test-slave-kk5c krb5kdc[14100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.253@KRBTEST.COM
I20260620 21:53:03.244724 14256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:53:03.245651 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.236326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:35073 (local address 127.0.0.1:33840)
0620 21:53:03.236569 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:03.236581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:03.236655 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:03.236877 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:03.236880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:03.237070 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:03.237174 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:03.237181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.238084 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.238087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:03.238927 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:03.238937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.239583 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.239586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:03.239652 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.240177 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:03.240192 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:03.241999 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:03.244098 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:03.244101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:03.244103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:03.244352 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:03.244631 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:03.244634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:03.244635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:03.244707 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:03.245157 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:03.245160 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:03.245236 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 21:53:03.245341 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 21:53:03.245478 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260620 21:53:03.246104 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.236557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.0.193.252:35073)
0620 21:53:03.236664 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:03.236668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:03.236682 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:53:03.236720 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:03.236722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:03.236763 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:03.236847 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:03.237289 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.237981 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.239035 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.239385 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.239485 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.242141 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:03.242147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:03.242152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:03.242178 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:03.243985 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.244466 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.244469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.244471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.244545 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.244796 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.244800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.244802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.245021 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:03.245117 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:03.245739 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 21:53:03.245918 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:53:03.248569 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.235616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.0.193.253:40905)
0620 21:53:03.236296 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:03.236301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:03.236352 (+ 51us) server_negotiation.cc:407] Connection header received
0620 21:53:03.237935 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:03.237938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:03.237985 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:03.238036 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:03.238924 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.239676 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.240833 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.241210 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.241288 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.244976 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:03.244979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:03.244981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:03.245005 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:03.246620 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.247325 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.247328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.247330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.247418 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.247739 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.247744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.247745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.247940 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:03.248166 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:03.248298 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 21:53:03.248409 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":91,"threads_started":1}
I20260620 21:53:03.249855 14331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a: Bootstrap starting.
I20260620 21:53:03.252029 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:53:03.252825 14331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:03.253718 14331 log.cc:824] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a: Log is configured to *not* fsync() on all Append() calls
W20260620 21:53:03.254652 14365 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:53:03.254746 14365 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:53:03.255385 14353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } } }
I20260620 21:53:03.255517 14353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:53:03.257598 14331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a: No bootstrap required, opened a new log
I20260620 21:53:03.256973 14188 master_runner.cc:186] Error getting master registration for 127.0.193.252:35073: OK, instance_id { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" instance_seqno: 1781992383187397 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:53:03.261055 14188 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:53:03.261883 14331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } }
I20260620 21:53:03.262156 14331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:03.262216 14331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ab9aaca87d42e0bab94e63269c2a9a, State: Initialized, Role: FOLLOWER
I20260620 21:53:03.264634 14331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } }
I20260620 21:53:03.266359 14331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:53:03.277539 14262 master_runner.cc:263] No leader master found from master 56ab9aaca87d42e0bab94e63269c2a9a
I20260620 21:53:03.277617 14262 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:53:03.277915 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.254894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.0.193.253:40905)
0620 21:53:03.255120 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:03.255124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:03.263485 (+ 8361us) server_negotiation.cc:407] Connection header received
0620 21:53:03.263602 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:03.263605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:03.263662 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:03.263740 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:03.264763 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.265638 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.266864 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.267276 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.267369 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.271042 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:03.271046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:03.271049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:03.271079 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:03.272713 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.276497 (+ 3784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.276502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.276504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.276602 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.276909 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.276913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.276914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.277151 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:03.277267 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:03.277630 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 21:53:03.277757 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260620 21:53:03.279500 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.256707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.0.193.252:35073)
0620 21:53:03.256827 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:03.256833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:03.263814 (+ 6981us) server_negotiation.cc:407] Connection header received
0620 21:53:03.267418 (+ 3604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:03.267421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:03.267479 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:03.267548 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:03.268560 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.269264 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.272224 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.272610 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.272690 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.275306 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:03.275309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:03.275313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:03.275342 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:03.277040 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.277542 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.277546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.277550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.277635 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.278349 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.278354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.278357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.278606 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:03.278738 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:03.279094 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 21:53:03.279263 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:53:03.280057 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:03.254746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.0.193.254:39275)
0620 21:53:03.254996 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:03.255003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:03.255292 (+ 289us) server_negotiation.cc:407] Connection header received
0620 21:53:03.255484 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:03.255487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:03.255535 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:03.255588 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:03.263336 (+ 7748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.264152 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.268971 (+ 4819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:03.269341 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:03.269425 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:03.273478 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:03.273484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:03.273489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:03.273522 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:03.275255 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.278667 (+ 3412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.278671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.278673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.278767 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:03.279098 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:03.279102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:03.279103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:03.279339 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:03.279451 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:03.279760 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 21:53:03.279887 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260620 21:53:03.280998 14375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } } }
I20260620 21:53:03.281138 14375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [sys.catalog]: This master's current role is: FOLLOWER
W20260620 21:53:03.289671 14386 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:53:03.289860 14386 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:53:03.326623 14353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:53:03.326843 14353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } }
I20260620 21:53:03.327986 14353 leader_election.cc:288] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6561c7a3110544268c23ea7da21bbb0d (127.0.193.254:39275), 56ab9aaca87d42e0bab94e63269c2a9a (127.0.193.252:35073)
I20260620 21:53:03.328691 14155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "251aa2d9b54a468d82abe1ac00918162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6561c7a3110544268c23ea7da21bbb0d" is_pre_election: true
I20260620 21:53:03.329041 14155 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 251aa2d9b54a468d82abe1ac00918162 in term 0.
I20260620 21:53:03.329484 14207 leader_election.cc:302] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 251aa2d9b54a468d82abe1ac00918162, 6561c7a3110544268c23ea7da21bbb0d; no voters:
I20260620 21:53:03.330250 14305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "251aa2d9b54a468d82abe1ac00918162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" is_pre_election: true
I20260620 21:53:03.330555 14305 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 251aa2d9b54a468d82abe1ac00918162 in term 0.
I20260620 21:53:03.331156 14353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:53:03.331277 14353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:53:03.331338 14353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:03.332641 14353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } }
I20260620 21:53:03.333067 14353 leader_election.cc:288] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [CANDIDATE]: Term 1 election: Requested vote from peers 6561c7a3110544268c23ea7da21bbb0d (127.0.193.254:39275), 56ab9aaca87d42e0bab94e63269c2a9a (127.0.193.252:35073)
I20260620 21:53:03.333621 14155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "251aa2d9b54a468d82abe1ac00918162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6561c7a3110544268c23ea7da21bbb0d"
I20260620 21:53:03.333753 14155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:03.335106 14155 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 251aa2d9b54a468d82abe1ac00918162 in term 1.
I20260620 21:53:03.335862 14305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "251aa2d9b54a468d82abe1ac00918162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ab9aaca87d42e0bab94e63269c2a9a"
I20260620 21:53:03.336014 14305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:03.337342 14305 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 251aa2d9b54a468d82abe1ac00918162 in term 1.
I20260620 21:53:03.335462 14207 leader_election.cc:302] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 251aa2d9b54a468d82abe1ac00918162, 6561c7a3110544268c23ea7da21bbb0d; no voters:
I20260620 21:53:03.341311 14353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:53:03.341435 14353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [term 1 LEADER]: Becoming Leader. State: Replica: 251aa2d9b54a468d82abe1ac00918162, State: Running, Role: LEADER
I20260620 21:53:03.341801 14353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } }
I20260620 21:53:03.343753 14353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 251aa2d9b54a468d82abe1ac00918162. Latest consensus state: current_term: 1 leader_uuid: "251aa2d9b54a468d82abe1ac00918162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } } }
I20260620 21:53:03.343871 14353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [sys.catalog]: This master's current role is: LEADER
I20260620 21:53:03.345885 14390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:53:03.346818 14390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:53:03.355373 14305 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [term 1 FOLLOWER]: Refusing update from remote peer 251aa2d9b54a468d82abe1ac00918162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:53:03.356092 14388 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:53:03.360577 14375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 251aa2d9b54a468d82abe1ac00918162. Latest consensus state: current_term: 1 leader_uuid: "251aa2d9b54a468d82abe1ac00918162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } } }
I20260620 21:53:03.360724 14375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:53:03.361889 14155 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [term 1 FOLLOWER]: Refusing update from remote peer 251aa2d9b54a468d82abe1ac00918162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:53:03.362533 14388 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:53:03.366909 14340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 251aa2d9b54a468d82abe1ac00918162. Latest consensus state: current_term: 1 leader_uuid: "251aa2d9b54a468d82abe1ac00918162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } } }
I20260620 21:53:03.367074 14340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:53:03.371613 14388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "251aa2d9b54a468d82abe1ac00918162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } } }
I20260620 21:53:03.371764 14388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [sys.catalog]: This master's current role is: LEADER
I20260620 21:53:03.373055 14375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "251aa2d9b54a468d82abe1ac00918162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } } }
I20260620 21:53:03.373188 14375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:53:03.373600 14391 mvcc.cc:205] Tried to move back new op lower bound from 7299040802215092224 to 7299040802171486208. Current Snapshot: MvccSnapshot[applied={T|T < 7299040802215092224}]
I20260620 21:53:03.376523 14353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "251aa2d9b54a468d82abe1ac00918162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } } }
I20260620 21:53:03.376662 14353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162 [sys.catalog]: This master's current role is: LEADER
I20260620 21:53:03.377882 14340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "251aa2d9b54a468d82abe1ac00918162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6561c7a3110544268c23ea7da21bbb0d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39275 } } peers { permanent_uuid: "251aa2d9b54a468d82abe1ac00918162" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40905 } } peers { permanent_uuid: "56ab9aaca87d42e0bab94e63269c2a9a" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 35073 } } }
I20260620 21:53:03.378038 14340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:53:03.381067 14394 mvcc.cc:205] Tried to move back new op lower bound from 7299040802215092224 to 7299040802171486208. Current Snapshot: MvccSnapshot[applied={T|T < 7299040802215092224}]
I20260620 21:53:03.382293 14390 catalog_manager.cc:1347] Generated new cluster ID: 9f9f966a4ef44ad2809a2e23ee2f10d1
I20260620 21:53:03.382367 14390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:53:03.412830 14390 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:53:03.415001 14390 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:53:03.442018 14390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 251aa2d9b54a468d82abe1ac00918162: Generated new TSK 0
I20260620 21:53:03.443028 14390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:53:04.244464 14351 catalog_manager.cc:1259] Loaded cluster ID: 9f9f966a4ef44ad2809a2e23ee2f10d1
I20260620 21:53:04.244581 14351 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d: loading cluster ID for follower catalog manager: success
I20260620 21:53:04.246861 14351 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d: acquiring CA information for follower catalog manager: success
I20260620 21:53:04.248231 14351 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6561c7a3110544268c23ea7da21bbb0d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:53:04.291302 14386 catalog_manager.cc:1259] Loaded cluster ID: 9f9f966a4ef44ad2809a2e23ee2f10d1
I20260620 21:53:04.291436 14386 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a: loading cluster ID for follower catalog manager: success
I20260620 21:53:04.293068 14386 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a: acquiring CA information for follower catalog manager: success
I20260620 21:53:04.294384 14386 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 56ab9aaca87d42e0bab94e63269c2a9a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:53:04.513898 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 14113
I20260620 21:53:04.520815 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 14188
I20260620 21:53:04.527097 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 14262
2026-06-20T21:53:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3229 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:53:04 dist-test-slave-kk5c krb5kdc[14404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:53:04 dist-test-slave-kk5c krb5kdc[14404](info): set up 2 sockets
Jun 20 21:53:04 dist-test-slave-kk5c krb5kdc[14404](info): commencing operation
krb5kdc: starting...
W20260620 21:53:05.640082 775 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:53:05 dist-test-slave-kk5c krb5kdc[14404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:53:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:53:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:53:05.779438 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:43891
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:43891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:53:05.907900 14417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:05.908211 14417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:05.908254 14417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:05.912534 14417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:53:05.912621 14417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:05.912640 14417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:05.912658 14417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:53:05.912675 14417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:53:05.918604 14417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:43891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:43891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:05.919857 14417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:05.921201 14417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:05.931535 14423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:05.931530 14425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:05.931530 14422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:05.932173 14417 server_base.cc:1034] running on GCE node
I20260620 21:53:05.932771 14417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:05.934017 14417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:05.935308 14417 hybrid_clock.cc:648] HybridClock initialized: now 1781992385935264 us; error 68 us; skew 500 ppm
Jun 20 21:53:05 dist-test-slave-kk5c krb5kdc[14404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:05.938526 14417 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:53:05.939633 14417 webserver.cc:460] Webserver started at http://127.0.193.254:43445/ using document root <none> and password file <none>
I20260620 21:53:05.940332 14417 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:05.940418 14417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:05.940603 14417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:53:05.942813 14417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "a5f5eb9e2f64410c8dd33f9e92aa9023"
format_stamp: "Formatted at 2026-06-20 21:53:05 on dist-test-slave-kk5c"
server_key: "06bfff8fd8f22262042646bd1c1cd742"
server_key_iv: "ed7d78de8b3401f83c8177482c786f04"
server_key_version: "encryptionkey@0"
I20260620 21:53:05.943457 14417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "a5f5eb9e2f64410c8dd33f9e92aa9023"
format_stamp: "Formatted at 2026-06-20 21:53:05 on dist-test-slave-kk5c"
server_key: "06bfff8fd8f22262042646bd1c1cd742"
server_key_iv: "ed7d78de8b3401f83c8177482c786f04"
server_key_version: "encryptionkey@0"
I20260620 21:53:05.947547 14417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:53:05.950176 14432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:05.951784 14417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:05.951946 14417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "a5f5eb9e2f64410c8dd33f9e92aa9023"
format_stamp: "Formatted at 2026-06-20 21:53:05 on dist-test-slave-kk5c"
server_key: "06bfff8fd8f22262042646bd1c1cd742"
server_key_iv: "ed7d78de8b3401f83c8177482c786f04"
server_key_version: "encryptionkey@0"
I20260620 21:53:05.952080 14417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:05.964423 14417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:05.968480 14417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:05.968665 14417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:05.975980 14417 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:43891
I20260620 21:53:05.975998 14484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:43891 every 8 connection(s)
I20260620 21:53:05.977044 14417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:53:05.977348 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 14417
I20260620 21:53:05.977489 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:53:05.977850 775 external_mini_cluster.cc:1428] Setting key 2c95d5a5f2d808482e0c6c973636fd68
I20260620 21:53:05.980386 14485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:05.987649 14485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023: Bootstrap starting.
Jun 20 21:53:05 dist-test-slave-kk5c krb5kdc[14404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:05.990872 14485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:05.991760 14485 log.cc:824] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:05.994328 14485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023: No bootstrap required, opened a new log
I20260620 21:53:05.996068 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:05.979435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48302 (local address 127.0.193.254:43891)
0620 21:53:05.979919 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:05.979936 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:05.979980 (+ 44us) server_negotiation.cc:407] Connection header received
0620 21:53:05.980754 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:05.980775 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:05.981153 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:05.981517 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:05.982357 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:05.983495 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:05.984745 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:05.985374 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:05.985759 (+ 385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:05.990136 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:05.990142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:05.990157 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:05.990196 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:05.992327 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:05.993074 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:05.993079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:05.993084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:05.993208 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:05.993734 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:05.993737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:05.993739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:05.994220 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:05.994516 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:05.994812 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 21:53:05.995178 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":148,"threads_started":1}
I20260620 21:53:05.998864 14485 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f5eb9e2f64410c8dd33f9e92aa9023" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43891 } }
I20260620 21:53:05.999157 14485 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:05.999279 14485 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f5eb9e2f64410c8dd33f9e92aa9023, State: Initialized, Role: FOLLOWER
I20260620 21:53:05.999430 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:53:05.999804 14485 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f5eb9e2f64410c8dd33f9e92aa9023" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43891 } }
I20260620 21:53:06.000077 14485 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:53:06.000144 14485 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:53:06.000253 14485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:06.001353 14485 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f5eb9e2f64410c8dd33f9e92aa9023" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43891 } }
I20260620 21:53:06.001739 14485 leader_election.cc:302] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5f5eb9e2f64410c8dd33f9e92aa9023; no voters:
I20260620 21:53:06.002061 14485 leader_election.cc:288] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:53:06.002328 14490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:53:06.002596 14490 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [term 1 LEADER]: Becoming Leader. State: Replica: a5f5eb9e2f64410c8dd33f9e92aa9023, State: Running, Role: LEADER
I20260620 21:53:06.002911 14490 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f5eb9e2f64410c8dd33f9e92aa9023" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43891 } }
I20260620 21:53:06.003726 14485 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:53:06.004742 14492 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5f5eb9e2f64410c8dd33f9e92aa9023. Latest consensus state: current_term: 1 leader_uuid: "a5f5eb9e2f64410c8dd33f9e92aa9023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f5eb9e2f64410c8dd33f9e92aa9023" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43891 } } }
I20260620 21:53:06.004812 14491 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5f5eb9e2f64410c8dd33f9e92aa9023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f5eb9e2f64410c8dd33f9e92aa9023" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 43891 } } }
I20260620 21:53:06.004895 14492 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [sys.catalog]: This master's current role is: LEADER
I20260620 21:53:06.005051 14491 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023 [sys.catalog]: This master's current role is: LEADER
W20260620 21:53:06.008831 14505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:53:06.008920 14505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:53:06.009011 14499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:53:06.009754 14499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:53:06.015672 14499 catalog_manager.cc:1347] Generated new cluster ID: 743c25d3f37241cfa1ede5d23e451444
I20260620 21:53:06.015774 14499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:53:06.026957 14499 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:53:06.028347 14499 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:53:06.035789 14499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5f5eb9e2f64410c8dd33f9e92aa9023: Generated new TSK 0
I20260620 21:53:06.036616 14499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:53:06.056239 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 14417
2026-06-20T21:53:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1526 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:53:06 dist-test-slave-kk5c krb5kdc[14512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:53:06 dist-test-slave-kk5c krb5kdc[14512](info): set up 2 sockets
Jun 20 21:53:06 dist-test-slave-kk5c krb5kdc[14512](info): commencing operation
krb5kdc: starting...
W20260620 21:53:07.131668 775 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:53:07 dist-test-slave-kk5c krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:53:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:53:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:53:07.272332 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:35605
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:35605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:53:07.396087 14525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:07.396452 14525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:07.396500 14525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:07.401314 14525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:53:07.401412 14525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:07.401436 14525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:07.401454 14525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:53:07.401470 14525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:53:07.407105 14525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:35605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:35605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:07.408326 14525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:07.409438 14525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:07.419059 14533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:07.419052 14530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:07.419052 14531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:07.419327 14525 server_base.cc:1034] running on GCE node
I20260620 21:53:07.419991 14525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:07.421123 14525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:07.422323 14525 hybrid_clock.cc:648] HybridClock initialized: now 1781992387422303 us; error 35 us; skew 500 ppm
Jun 20 21:53:07 dist-test-slave-kk5c krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:07.430471 14525 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:53:07.431917 14525 webserver.cc:460] Webserver started at http://127.0.193.254:41817/ using document root <none> and password file <none>
I20260620 21:53:07.432561 14525 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:07.432662 14525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:07.432914 14525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:53:07.435075 14525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "24805f458774465298e06cac6c76bfbb"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "92a1e4c223c725e4c66100ea1219fa19"
server_key_iv: "e616f0ccb13c03cbd3a491c38655cd47"
server_key_version: "encryptionkey@0"
I20260620 21:53:07.435765 14525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "24805f458774465298e06cac6c76bfbb"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "92a1e4c223c725e4c66100ea1219fa19"
server_key_iv: "e616f0ccb13c03cbd3a491c38655cd47"
server_key_version: "encryptionkey@0"
I20260620 21:53:07.439949 14525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:53:07.442670 14540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:07.444337 14525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:07.444474 14525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "24805f458774465298e06cac6c76bfbb"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "92a1e4c223c725e4c66100ea1219fa19"
server_key_iv: "e616f0ccb13c03cbd3a491c38655cd47"
server_key_version: "encryptionkey@0"
I20260620 21:53:07.444626 14525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:07.468922 14525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:07.472426 14525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:07.472791 14525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:07.480110 14525 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:35605
I20260620 21:53:07.480448 14592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:35605 every 8 connection(s)
I20260620 21:53:07.481173 14525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:53:07.484179 14593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:07.490361 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 14525
I20260620 21:53:07.490541 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:53:07.490752 14593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Bootstrap starting.
I20260620 21:53:07.490834 775 external_mini_cluster.cc:1428] Setting key b88bcee809ed0fceec4b2ac03833d033
I20260620 21:53:07.493505 14593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:07.494380 14593 log.cc:824] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:07.496837 14593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: No bootstrap required, opened a new log
Jun 20 21:53:07 dist-test-slave-kk5c krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:07.501418 14593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } }
I20260620 21:53:07.501698 14593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:07.501787 14593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24805f458774465298e06cac6c76bfbb, State: Initialized, Role: FOLLOWER
I20260620 21:53:07.502305 14593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } }
I20260620 21:53:07.502468 14593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:53:07.502550 14593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:53:07.502657 14593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:07.503816 14593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } }
I20260620 21:53:07.504259 14593 leader_election.cc:302] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24805f458774465298e06cac6c76bfbb; no voters:
I20260620 21:53:07.504674 14593 leader_election.cc:288] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:53:07.504753 14598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:53:07.504974 14598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 1 LEADER]: Becoming Leader. State: Replica: 24805f458774465298e06cac6c76bfbb, State: Running, Role: LEADER
I20260620 21:53:07.505345 14598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } }
I20260620 21:53:07.506273 14593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:53:07.507301 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:07.492198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.0.193.254:35605)
0620 21:53:07.492688 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:07.492704 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:07.492740 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:53:07.493418 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:07.493436 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:07.493832 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:07.494134 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:07.495084 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:07.496236 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:07.497418 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:07.497984 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:07.498150 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:07.501206 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:07.501212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:07.501263 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:07.501308 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:07.503620 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:07.504380 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:07.504389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:07.504401 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:07.504568 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:07.504902 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:07.504906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:07.504908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:07.505490 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:07.505702 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:07.506007 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 21:53:07.506332 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":198,"threads_started":1}
I20260620 21:53:07.507653 14600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24805f458774465298e06cac6c76bfbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } } }
I20260620 21:53:07.507923 14600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: This master's current role is: LEADER
I20260620 21:53:07.508409 14599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24805f458774465298e06cac6c76bfbb. Latest consensus state: current_term: 1 leader_uuid: "24805f458774465298e06cac6c76bfbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } } }
I20260620 21:53:07.508529 14599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: This master's current role is: LEADER
W20260620 21:53:07.512084 14613 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:53:07.512172 14613 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:53:07.512266 14607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:53:07.513113 14607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:53:07.519091 14607 catalog_manager.cc:1347] Generated new cluster ID: fe7cda4d4e50402cbe0db6b3a4965fbc
I20260620 21:53:07.519302 14607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:53:07.529776 14607 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:53:07.530800 14607 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:53:07.540392 14607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Generated new TSK 0
I20260620 21:53:07.541361 14607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:53:07.582474 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:35605
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:53:07.719681 14619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:07.720005 14619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:07.720052 14619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:07.724466 14619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:07.724553 14619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:07.724618 14619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:53:07.730648 14619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:07.731953 14619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:07.733062 14619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:07.743880 14624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:07.743880 14625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:07.743896 14627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:07.744074 14619 server_base.cc:1034] running on GCE node
I20260620 21:53:07.744864 14619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:07.745628 14619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:07.746824 14619 hybrid_clock.cc:648] HybridClock initialized: now 1781992387746793 us; error 59 us; skew 500 ppm
Jun 20 21:53:07 dist-test-slave-kk5c krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:07.750226 14619 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:53:07.751390 14619 webserver.cc:460] Webserver started at http://127.0.193.193:41259/ using document root <none> and password file <none>
I20260620 21:53:07.752017 14619 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:07.752120 14619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:07.752364 14619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:53:07.754590 14619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "e19037361c4a449f88e644722ff65f3b"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "fd5aa02cdb02a9c9bb30b7b27d2b96aa"
server_key_iv: "bab7344b5aa1194dc4f38dbe22276e4d"
server_key_version: "encryptionkey@0"
I20260620 21:53:07.755313 14619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "e19037361c4a449f88e644722ff65f3b"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "fd5aa02cdb02a9c9bb30b7b27d2b96aa"
server_key_iv: "bab7344b5aa1194dc4f38dbe22276e4d"
server_key_version: "encryptionkey@0"
I20260620 21:53:07.759891 14619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:53:07.762673 14634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:07.764320 14619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:07.764492 14619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "e19037361c4a449f88e644722ff65f3b"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "fd5aa02cdb02a9c9bb30b7b27d2b96aa"
server_key_iv: "bab7344b5aa1194dc4f38dbe22276e4d"
server_key_version: "encryptionkey@0"
I20260620 21:53:07.764631 14619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:07.775671 14619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:07.779170 14619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:07.779422 14619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:07.780054 14619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:07.780918 14619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:53:07.780997 14619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:07.781086 14619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:53:07.781139 14619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:07.790355 14619 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:33441
I20260620 21:53:07.790391 14747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:33441 every 8 connection(s)
I20260620 21:53:07.791570 14619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:53:07.801856 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 14619
I20260620 21:53:07.802009 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
Jun 20 21:53:07 dist-test-slave-kk5c krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:07.802373 775 external_mini_cluster.cc:1428] Setting key d7708a06f12883e3911a9d985701bc80
I20260620 21:53:07.808063 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:07.793700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:59843 (local address 127.0.193.254:35605)
0620 21:53:07.794103 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:07.794107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:07.795145 (+ 1038us) server_negotiation.cc:407] Connection header received
0620 21:53:07.796152 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:07.796156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:07.796206 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:07.796358 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:07.797817 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:07.798401 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:07.799747 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:07.800011 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:07.800104 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:07.803002 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:07.803006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:07.803009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:07.803040 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:07.805161 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:07.805878 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:07.805886 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:07.805891 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:07.805997 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:07.806518 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:07.806520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:07.806522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:07.806828 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:07.806919 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:07.807774 (+ 855us) server_negotiation.cc:299] Negotiation successful
0620 21:53:07.807910 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296}
I20260620 21:53:07.809319 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:07.794138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:59843)
0620 21:53:07.794964 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:07.795023 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:07.795915 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:07.796624 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:07.796631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:07.797111 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:07.797636 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:07.797651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:07.798586 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:07.798589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:07.799609 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:07.799619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:07.800185 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:07.800188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:07.800348 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:07.800921 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:07.800943 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:07.802691 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:07.805343 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:07.805349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:07.805366 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:07.805731 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:07.806149 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:07.806155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:07.806158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:07.806401 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:07.806958 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:07.806967 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:07.807426 (+ 459us) client_negotiation.cc:769] Sending connection context
0620 21:53:07.807685 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 21:53:07.808112 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":144,"threads_started":1}
I20260620 21:53:07.811070 14748 heartbeater.cc:347] Connected to a master server at 127.0.193.254:35605
I20260620 21:53:07.811394 14748 heartbeater.cc:464] Registering TS with master...
I20260620 21:53:07.811995 14748 heartbeater.cc:511] Master 127.0.193.254:35605 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
I20260620 21:53:07.813864 14553 ts_manager.cc:194] Registered new tserver with Master: e19037361c4a449f88e644722ff65f3b (127.0.193.193:33441)
I20260620 21:53:07.815459 14553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:59843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:53:07.836015 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:35605
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:53:07.955873 14753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:07.956220 14753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:07.956334 14753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:07.960681 14753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:07.960789 14753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:07.960889 14753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:53:07.967007 14753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:07.968299 14753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:07.969406 14753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:07.979624 14758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:07.979729 14759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:07.979800 14753 server_base.cc:1034] running on GCE node
W20260620 21:53:07.979743 14761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:07.980412 14753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:07.981113 14753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:07.982302 14753 hybrid_clock.cc:648] HybridClock initialized: now 1781992387982285 us; error 34 us; skew 500 ppm
Jun 20 21:53:07 dist-test-slave-kk5c krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:07.985574 14753 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:53:07.986676 14753 webserver.cc:460] Webserver started at http://127.0.193.194:45283/ using document root <none> and password file <none>
I20260620 21:53:07.987412 14753 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:07.987524 14753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:07.987782 14753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:53:07.990293 14753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "fa52ccbd3c1045ed9caaaf7b025c686d"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "9e95dd4cd283d2ee21d477c8fa568744"
server_key_iv: "b23038021fa6d09c7b8e2f9ea62f9f1e"
server_key_version: "encryptionkey@0"
I20260620 21:53:07.990926 14753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "fa52ccbd3c1045ed9caaaf7b025c686d"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "9e95dd4cd283d2ee21d477c8fa568744"
server_key_iv: "b23038021fa6d09c7b8e2f9ea62f9f1e"
server_key_version: "encryptionkey@0"
I20260620 21:53:07.995641 14753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:53:07.998605 14768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:08.000434 14753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:08.000650 14753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "fa52ccbd3c1045ed9caaaf7b025c686d"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "9e95dd4cd283d2ee21d477c8fa568744"
server_key_iv: "b23038021fa6d09c7b8e2f9ea62f9f1e"
server_key_version: "encryptionkey@0"
I20260620 21:53:08.000799 14753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:08.019109 14753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:08.022770 14753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:08.022987 14753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:08.023703 14753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:08.024585 14753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:53:08.024662 14753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:08.024727 14753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:53:08.024780 14753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:08.034377 14753 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:40759
I20260620 21:53:08.034399 14881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:40759 every 8 connection(s)
I20260620 21:53:08.035372 14753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:53:08.044034 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 14753
I20260620 21:53:08.044171 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:53:08.044520 775 external_mini_cluster.cc:1428] Setting key b4bff766f8a9f8c40bfe5de2d07cad6e
Jun 20 21:53:08 dist-test-slave-kk5c krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:08.052738 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.037598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:56301 (local address 127.0.193.254:35605)
0620 21:53:08.037787 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:08.037791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:08.039020 (+ 1229us) server_negotiation.cc:407] Connection header received
0620 21:53:08.040243 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:08.040247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:08.040310 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:08.040421 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:08.041751 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.042341 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.043759 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.044177 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.044308 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.047687 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:08.047694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:08.047700 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:08.047735 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:08.049898 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:08.050655 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:08.050659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:08.050660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:08.050743 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:08.051317 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:08.051322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:08.051325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:08.051585 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:08.051672 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:08.052472 (+ 800us) server_negotiation.cc:299] Negotiation successful
0620 21:53:08.052594 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 21:53:08.054057 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.038030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:56301)
0620 21:53:08.038828 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:08.038884 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:08.039999 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:08.040609 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:08.040617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:08.041050 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:08.041587 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.041601 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.042535 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.042538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.043586 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.043597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.044360 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.044363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.044523 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.045328 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:08.045345 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:08.047501 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:08.050055 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:08.050066 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:08.050092 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:08.050509 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:08.050852 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:08.050857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:08.050862 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:08.051078 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:08.051713 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:08.051722 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:08.052116 (+ 394us) client_negotiation.cc:769] Sending connection context
0620 21:53:08.052379 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 21:53:08.052819 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":127,"threads_started":1}
I20260620 21:53:08.055573 14882 heartbeater.cc:347] Connected to a master server at 127.0.193.254:35605
I20260620 21:53:08.055869 14882 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
I20260620 21:53:08.056468 14882 heartbeater.cc:511] Master 127.0.193.254:35605 requested a full tablet report, sending...
I20260620 21:53:08.057607 14553 ts_manager.cc:194] Registered new tserver with Master: fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194:40759)
I20260620 21:53:08.058274 14553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:56301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:53:08.076695 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:35605
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:53:08.199702 14887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:08.200078 14887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:08.200161 14887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:08.204444 14887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:08.204559 14887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:08.204658 14887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:53:08.210956 14887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:08.212180 14887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:08.213264 14887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:08.224844 14895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:08.224828 14893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:08.224828 14892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:08.225054 14887 server_base.cc:1034] running on GCE node
I20260620 21:53:08.225879 14887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:08.226753 14887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:08.228003 14887 hybrid_clock.cc:648] HybridClock initialized: now 1781992388227970 us; error 39 us; skew 500 ppm
Jun 20 21:53:08 dist-test-slave-kk5c krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:08.232847 14887 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:53:08.234177 14887 webserver.cc:460] Webserver started at http://127.0.193.195:36685/ using document root <none> and password file <none>
I20260620 21:53:08.234851 14887 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:08.234925 14887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:08.235178 14887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:53:08.237646 14887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "eabe2a80359f42fab4c94dba012ad0bd"
format_stamp: "Formatted at 2026-06-20 21:53:08 on dist-test-slave-kk5c"
server_key: "a0c23a279ed342e1954c66e097556ada"
server_key_iv: "f4e759157779b639e3c0d091ded09691"
server_key_version: "encryptionkey@0"
I20260620 21:53:08.238466 14887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "eabe2a80359f42fab4c94dba012ad0bd"
format_stamp: "Formatted at 2026-06-20 21:53:08 on dist-test-slave-kk5c"
server_key: "a0c23a279ed342e1954c66e097556ada"
server_key_iv: "f4e759157779b639e3c0d091ded09691"
server_key_version: "encryptionkey@0"
I20260620 21:53:08.244130 14887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 21:53:08.247314 14902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:08.248939 14887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:53:08.249106 14887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "eabe2a80359f42fab4c94dba012ad0bd"
format_stamp: "Formatted at 2026-06-20 21:53:08 on dist-test-slave-kk5c"
server_key: "a0c23a279ed342e1954c66e097556ada"
server_key_iv: "f4e759157779b639e3c0d091ded09691"
server_key_version: "encryptionkey@0"
I20260620 21:53:08.249250 14887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:08.267616 14887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:08.271958 14887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:08.272130 14887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:08.272696 14887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:08.273532 14887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:53:08.273581 14887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:08.273620 14887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:53:08.273636 14887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:08.284108 14887 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:37413
I20260620 21:53:08.284111 15015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:37413 every 8 connection(s)
I20260620 21:53:08.285163 14887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:53:08.294540 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 14887
I20260620 21:53:08.294667 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:53:08.294948 775 external_mini_cluster.cc:1428] Setting key 8ae8100db4f968cbbf664ccabd7f40f0
Jun 20 21:53:08 dist-test-slave-kk5c krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:08.300789 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.287245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:45165 (local address 127.0.193.254:35605)
0620 21:53:08.287457 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:08.287461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:08.288413 (+ 952us) server_negotiation.cc:407] Connection header received
0620 21:53:08.289692 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:08.289695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:08.289755 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:08.289895 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:08.291130 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.291884 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.293088 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.293326 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.293447 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.296806 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:08.296809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:08.296811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:08.296836 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:08.298413 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:08.299000 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:08.299003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:08.299005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:08.299103 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:08.299639 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:08.299642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:08.299643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:08.299842 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:08.299951 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:08.300549 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 21:53:08.300660 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 21:53:08.301761 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.287648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:45165)
0620 21:53:08.288203 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:08.288247 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:08.289471 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:08.290078 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:08.290085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:08.290468 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:08.290979 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.290994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.292024 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.292030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.292956 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.292965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.293491 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.293495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.293673 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.294378 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:08.294400 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:08.296636 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:08.298558 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:08.298564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:08.298578 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:08.298878 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:08.299315 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:08.299321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:08.299378 (+ 57us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:08.299536 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:08.300002 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:08.300008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:08.300279 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 21:53:08.300464 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 21:53:08.300765 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":181,"threads_started":1}
I20260620 21:53:08.303145 15016 heartbeater.cc:347] Connected to a master server at 127.0.193.254:35605
I20260620 21:53:08.303442 15016 heartbeater.cc:464] Registering TS with master...
I20260620 21:53:08.303921 15016 heartbeater.cc:511] Master 127.0.193.254:35605 requested a full tablet report, sending...
I20260620 21:53:08.305022 14553 ts_manager.cc:194] Registered new tserver with Master: eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413)
I20260620 21:53:08.305868 14553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:45165
I20260620 21:53:08.310596 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:53:08.321537 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.313071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44870 (local address 127.0.193.254:35605)
0620 21:53:08.313269 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:08.313274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:08.313424 (+ 150us) server_negotiation.cc:407] Connection header received
0620 21:53:08.313601 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:08.313605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:08.313648 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:08.313734 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:08.314542 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.315099 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.316254 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.316496 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.316609 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.318126 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:08.318134 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:08.318136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:08.318161 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:08.319724 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:08.320212 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:08.320216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:08.320218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:08.320300 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:08.320590 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:08.320594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:08.320596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:08.320838 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:08.320981 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:08.321230 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 21:53:08.321347 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 21:53:08.325111 14553 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:53:08.328269 14553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:53:08.344825 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.339379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.0.193.195:37413)
0620 21:53:08.339735 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:08.339749 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:08.339771 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:53:08.339848 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:08.339853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:08.340034 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:08.340174 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:08.340875 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.341941 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.343136 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.344032 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.344156 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.344317 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:08.344542 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 21:53:08.344654 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20260620 21:53:08.344854 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.339035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:37413 (local address 127.0.0.1:49992)
0620 21:53:08.339409 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:08.339423 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:08.339494 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:08.340201 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:08.340203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:08.340216 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:08.340356 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.340364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.342076 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.342082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.342980 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.343004 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.344194 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.344198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.344317 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.344362 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 21:53:08.344483 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 21:53:08.344624 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":26,"thread_start_us":89,"threads_started":1}
I20260620 21:53:08.345515 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.338538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48482 (local address 127.0.193.193:33441)
0620 21:53:08.338831 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:08.338839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:08.339255 (+ 416us) server_negotiation.cc:407] Connection header received
0620 21:53:08.339884 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:08.339889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:08.340036 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:08.340386 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:08.340837 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.341606 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.343586 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.344439 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.344577 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.344711 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:08.345263 (+ 552us) server_negotiation.cc:299] Negotiation successful
0620 21:53:08.345394 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260620 21:53:08.346189 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.338274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:33441 (local address 127.0.0.1:48482)
0620 21:53:08.339076 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:08.339135 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:08.339707 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:08.340588 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:08.340591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:08.340609 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:08.340698 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.340703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.341753 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.341757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.343357 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.343367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.344780 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.344783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.345070 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.345107 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 21:53:08.345939 (+ 832us) client_negotiation.cc:240] Negotiation successful
0620 21:53:08.346029 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":170,"thread_start_us":124,"threads_started":1}
I20260620 21:53:08.346571 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.338247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:40759 (local address 127.0.0.1:34924)
0620 21:53:08.339076 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:08.339134 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:08.339372 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:08.340120 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:08.340126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:08.340151 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:08.340361 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.340366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.341836 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.341838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.344338 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.344346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.345810 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.345813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.345909 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.346257 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 21:53:08.346337 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 21:53:08.346411 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":1257,"thread_start_us":102,"threads_started":1}
I20260620 21:53:08.346951 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.338362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.0.193.194:40759)
0620 21:53:08.339077 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:08.339089 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:08.339255 (+ 166us) server_negotiation.cc:407] Connection header received
0620 21:53:08.339580 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:08.339586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:08.339857 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:08.340009 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:08.340491 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.341703 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.344509 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.345510 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.345621 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.345749 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:08.346738 (+ 989us) server_negotiation.cc:299] Negotiation successful
0620 21:53:08.346843 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":118,"threads_started":1}
I20260620 21:53:08.348615 14950 tablet_service.cc:1467] Processing CreateTablet for tablet 14781f4967c64354bc5aae8af4077e06 (DEFAULT_TABLE table=test-table [id=b2bac02183c14cb791ff6be4a9a040e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:53:08.349154 14682 tablet_service.cc:1467] Processing CreateTablet for tablet 14781f4967c64354bc5aae8af4077e06 (DEFAULT_TABLE table=test-table [id=b2bac02183c14cb791ff6be4a9a040e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:53:08.349881 14950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14781f4967c64354bc5aae8af4077e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:08.350194 14682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14781f4967c64354bc5aae8af4077e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:08.350028 14816 tablet_service.cc:1467] Processing CreateTablet for tablet 14781f4967c64354bc5aae8af4077e06 (DEFAULT_TABLE table=test-table [id=b2bac02183c14cb791ff6be4a9a040e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:53:08.351104 14816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14781f4967c64354bc5aae8af4077e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:08.357568 15033 tablet_bootstrap.cc:492] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d: Bootstrap starting.
I20260620 21:53:08.359056 15035 tablet_bootstrap.cc:492] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd: Bootstrap starting.
I20260620 21:53:08.359316 15034 tablet_bootstrap.cc:492] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Bootstrap starting.
I20260620 21:53:08.360970 15033 tablet_bootstrap.cc:654] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:08.361387 15034 tablet_bootstrap.cc:654] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:08.361387 15035 tablet_bootstrap.cc:654] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:08.361944 15033 log.cc:824] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:08.362511 15034 log.cc:824] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:08.362804 15035 log.cc:824] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:08.364483 15034 tablet_bootstrap.cc:492] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: No bootstrap required, opened a new log
I20260620 21:53:08.364506 15033 tablet_bootstrap.cc:492] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d: No bootstrap required, opened a new log
I20260620 21:53:08.364712 15034 ts_tablet_manager.cc:1397] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:53:08.364714 15033 ts_tablet_manager.cc:1397] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 21:53:08.364818 15035 tablet_bootstrap.cc:492] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd: No bootstrap required, opened a new log
I20260620 21:53:08.364990 15035 ts_tablet_manager.cc:1397] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:53:08.368640 15034 raft_consensus.cc:348] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } } peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } }
I20260620 21:53:08.368968 15034 raft_consensus.cc:374] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:08.369027 15034 raft_consensus.cc:729] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19037361c4a449f88e644722ff65f3b, State: Initialized, Role: FOLLOWER
I20260620 21:53:08.368968 15033 raft_consensus.cc:348] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } } peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } }
I20260620 21:53:08.368968 15035 raft_consensus.cc:348] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } } peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } }
I20260620 21:53:08.369235 15035 raft_consensus.cc:374] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:08.369307 15035 raft_consensus.cc:729] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eabe2a80359f42fab4c94dba012ad0bd, State: Initialized, Role: FOLLOWER
I20260620 21:53:08.369340 15033 raft_consensus.cc:374] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:08.369405 15033 raft_consensus.cc:729] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa52ccbd3c1045ed9caaaf7b025c686d, State: Initialized, Role: FOLLOWER
I20260620 21:53:08.369665 15034 consensus_queue.cc:260] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } } peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } }
I20260620 21:53:08.369765 15033 consensus_queue.cc:260] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } } peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } }
I20260620 21:53:08.369765 15035 consensus_queue.cc:260] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } } peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } }
I20260620 21:53:08.370410 14748 heartbeater.cc:503] Master 127.0.193.254:35605 was elected leader, sending a full tablet report...
I20260620 21:53:08.370755 15034 ts_tablet_manager.cc:1428] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:53:08.370754 15033 ts_tablet_manager.cc:1428] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:53:08.370754 15035 ts_tablet_manager.cc:1428] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:53:08.371080 14882 heartbeater.cc:503] Master 127.0.193.254:35605 was elected leader, sending a full tablet report...
I20260620 21:53:08.371079 15016 heartbeater.cc:503] Master 127.0.193.254:35605 was elected leader, sending a full tablet report...
W20260620 21:53:08.536268 15017 tablet.cc:2369] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:53:08.536952 14883 tablet.cc:2369] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:53:08.543057 14749 tablet.cc:2369] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:53:08.730516 15041 raft_consensus.cc:482] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:53:08.730806 15041 raft_consensus.cc:504] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } } peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } }
I20260620 21:53:08.732122 15041 leader_election.cc:288] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e19037361c4a449f88e644722ff65f3b (127.0.193.193:33441), eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413)
I20260620 21:53:08.736867 15040 raft_consensus.cc:482] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:53:08.736943 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.732991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:52501 (local address 127.0.193.193:33441)
0620 21:53:08.733137 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:08.733144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:08.733244 (+ 100us) server_negotiation.cc:407] Connection header received
0620 21:53:08.733469 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:08.733472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:08.733524 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:08.733614 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:08.734211 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.734748 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.735885 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.736344 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.736483 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.736564 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:08.736730 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 21:53:08.736801 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 21:53:08.737084 15040 raft_consensus.cc:504] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } } peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } }
I20260620 21:53:08.736961 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.732584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:33441 (local address 127.0.193.194:52501)
0620 21:53:08.733115 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:08.733132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:08.733300 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:08.733629 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:08.733632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:08.733649 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:08.733977 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.733994 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.734876 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.734880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.735744 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.735753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.736481 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.736485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.736591 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.736623 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 21:53:08.736713 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 21:53:08.736795 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":181,"spinlock_wait_cycles":25472,"thread_start_us":92,"threads_started":1}
I20260620 21:53:08.737700 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.732979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:60433 (local address 127.0.193.195:37413)
0620 21:53:08.733116 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:08.733120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:08.733137 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:53:08.733457 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:08.733464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:08.733533 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:08.733651 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:08.734397 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.735380 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.736485 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.737072 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.737193 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.737241 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:08.737505 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 21:53:08.737582 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:53:08.737941 14702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14781f4967c64354bc5aae8af4077e06" candidate_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19037361c4a449f88e644722ff65f3b" is_pre_election: true
I20260620 21:53:08.738122 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.732584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:37413 (local address 127.0.193.194:60433)
0620 21:53:08.733014 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:08.733032 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:08.733159 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:08.733659 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:08.733662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:08.733710 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:08.734204 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.734215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.735561 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.735566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.736346 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.736355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.737217 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.737224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.737346 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.737373 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:53:08.737862 (+ 489us) client_negotiation.cc:240] Negotiation successful
0620 21:53:08.737950 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"spinlock_wait_cycles":2048,"thread_start_us":107,"threads_started":1}
I20260620 21:53:08.738248 14702 raft_consensus.cc:2454] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa52ccbd3c1045ed9caaaf7b025c686d in term 0.
I20260620 21:53:08.738755 14970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14781f4967c64354bc5aae8af4077e06" candidate_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eabe2a80359f42fab4c94dba012ad0bd" is_pre_election: true
I20260620 21:53:08.738823 14771 leader_election.cc:302] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e19037361c4a449f88e644722ff65f3b, fa52ccbd3c1045ed9caaaf7b025c686d; no voters:
I20260620 21:53:08.738961 14970 raft_consensus.cc:2454] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa52ccbd3c1045ed9caaaf7b025c686d in term 0.
I20260620 21:53:08.739235 15040 leader_election.cc:288] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194:40759), e19037361c4a449f88e644722ff65f3b (127.0.193.193:33441)
I20260620 21:53:08.739529 15041 raft_consensus.cc:2781] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:53:08.739643 15041 raft_consensus.cc:482] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:53:08.739709 15041 raft_consensus.cc:3037] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:08.741099 15041 raft_consensus.cc:504] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } } peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } }
I20260620 21:53:08.741823 14970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14781f4967c64354bc5aae8af4077e06" candidate_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eabe2a80359f42fab4c94dba012ad0bd"
I20260620 21:53:08.741963 14970 raft_consensus.cc:3037] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:08.743047 14970 raft_consensus.cc:2454] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa52ccbd3c1045ed9caaaf7b025c686d in term 1.
I20260620 21:53:08.743252 15044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.738722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:33441 (local address 127.0.193.195:47059)
0620 21:53:08.739493 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:08.739507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:08.739647 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:08.739901 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:08.739904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:08.739918 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:08.740038 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.740044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.740904 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.740907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.742111 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.742121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.742776 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.742779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.742891 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.742914 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:53:08.742992 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 21:53:08.743064 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":26,"thread_start_us":151,"threads_started":1}
I20260620 21:53:08.743296 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.739318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:47059 (local address 127.0.193.193:33441)
0620 21:53:08.739456 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:08.739459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:08.739578 (+ 119us) server_negotiation.cc:407] Connection header received
0620 21:53:08.739739 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:08.739744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:08.739792 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:08.739880 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:08.740155 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.740767 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.742232 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.742652 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.742783 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.742836 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:08.743066 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 21:53:08.743154 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:53:08.743484 14772 leader_election.cc:302] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eabe2a80359f42fab4c94dba012ad0bd, fa52ccbd3c1045ed9caaaf7b025c686d; no voters:
I20260620 21:53:08.743306 14702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14781f4967c64354bc5aae8af4077e06" candidate_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19037361c4a449f88e644722ff65f3b"
I20260620 21:53:08.743675 14702 raft_consensus.cc:3037] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:08.743724 14701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14781f4967c64354bc5aae8af4077e06" candidate_uuid: "eabe2a80359f42fab4c94dba012ad0bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19037361c4a449f88e644722ff65f3b" is_pre_election: true
I20260620 21:53:08.744132 15041 leader_election.cc:288] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [CANDIDATE]: Term 1 election: Requested vote from peers e19037361c4a449f88e644722ff65f3b (127.0.193.193:33441), eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413)
I20260620 21:53:08.744244 15041 raft_consensus.cc:2781] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:53:08.744309 15041 raft_consensus.cc:686] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [term 1 LEADER]: Becoming Leader. State: Replica: fa52ccbd3c1045ed9caaaf7b025c686d, State: Running, Role: LEADER
I20260620 21:53:08.744688 15041 consensus_queue.cc:237] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } } peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } }
I20260620 21:53:08.745041 14702 raft_consensus.cc:2454] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa52ccbd3c1045ed9caaaf7b025c686d in term 1.
I20260620 21:53:08.745693 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.738664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:40759 (local address 127.0.193.195:42097)
0620 21:53:08.739144 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:08.739158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:08.739332 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:08.739698 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:08.739701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:08.739723 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:08.739848 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.739855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.740795 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.740798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.742108 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:08.742118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.745252 (+ 3134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.745256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:08.745373 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.745396 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:53:08.745483 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 21:53:08.745569 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":88}
I20260620 21:53:08.745750 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.738900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:42097 (local address 127.0.193.194:40759)
0620 21:53:08.739179 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:08.739209 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:08.739226 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:53:08.739460 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:08.739464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:08.739519 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:08.739588 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:08.739976 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.740664 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.742892 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.743565 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.743690 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.743757 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:08.745518 (+ 1761us) server_negotiation.cc:299] Negotiation successful
0620 21:53:08.745622 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260620 21:53:08.746315 14836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14781f4967c64354bc5aae8af4077e06" candidate_uuid: "eabe2a80359f42fab4c94dba012ad0bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" is_pre_election: true
I20260620 21:53:08.746992 14906 leader_election.cc:302] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eabe2a80359f42fab4c94dba012ad0bd; no voters: e19037361c4a449f88e644722ff65f3b, fa52ccbd3c1045ed9caaaf7b025c686d
I20260620 21:53:08.747511 15040 raft_consensus.cc:2726] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 21:53:08.748528 14556 catalog_manager.cc:5495] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d reported cstate change: term changed from 0 to 1, leader changed from <none> to fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194). New cstate: current_term: 1 leader_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } health_report { overall_health: UNKNOWN } } }
I20260620 21:53:08.800841 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.796750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.0.193.194:40759)
0620 21:53:08.796920 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:08.796924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:08.796938 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:53:08.797057 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:08.797059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:08.797113 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:08.797205 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:53:08.797456 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.797963 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.799143 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:08.799455 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:08.799562 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:08.799912 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:53:08.800392 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:53:08.800498 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:08.800594 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 21:53:08.800648 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 21:53:08.807771 14702 raft_consensus.cc:1261] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Refusing update from remote peer fa52ccbd3c1045ed9caaaf7b025c686d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:53:08.807822 14970 raft_consensus.cc:1261] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [term 1 FOLLOWER]: Refusing update from remote peer fa52ccbd3c1045ed9caaaf7b025c686d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:53:08.808631 15041 consensus_queue.cc:1035] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:53:08.809154 15045 consensus_queue.cc:1035] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:53:08.818696 15050 mvcc.cc:205] Tried to move back new op lower bound from 7299040824546672640 to 7299040824301707264. Current Snapshot: MvccSnapshot[applied={T|T < 7299040824546672640}]
I20260620 21:53:08.821085 15049 mvcc.cc:205] Tried to move back new op lower bound from 7299040824546672640 to 7299040824301707264. Current Snapshot: MvccSnapshot[applied={T|T < 7299040824546672640}]
I20260620 21:53:08.829274 14553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:53:08.829492 14553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:53:08.832382 14553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Sending DeleteTablet for 3 replicas of tablet 14781f4967c64354bc5aae8af4077e06
I20260620 21:53:08.833412 14682 tablet_service.cc:1514] Processing DeleteTablet for tablet 14781f4967c64354bc5aae8af4077e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:08 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:48482
I20260620 21:53:08.833422 14816 tablet_service.cc:1514] Processing DeleteTablet for tablet 14781f4967c64354bc5aae8af4077e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:08 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:34924
I20260620 21:53:08.833576 14950 tablet_service.cc:1514] Processing DeleteTablet for tablet 14781f4967c64354bc5aae8af4077e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:08 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:49992
I20260620 21:53:08.834034 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 14619
I20260620 21:53:08.834141 15060 tablet_replica.cc:330] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd: stopping tablet replica
I20260620 21:53:08.834275 15059 tablet_replica.cc:330] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: stopping tablet replica
I20260620 21:53:08.834535 15060 raft_consensus.cc:2229] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:53:08.834822 15058 tablet_replica.cc:330] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d: stopping tablet replica
I20260620 21:53:08.834913 15060 raft_consensus.cc:2258] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:53:08.835274 15058 raft_consensus.cc:2229] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:53:08.835721 15058 raft_consensus.cc:2258] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:53:08.836757 15060 ts_tablet_manager.cc:1905] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:53:08.838176 15058 ts_tablet_manager.cc:1905] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:53:08.840619 15060 ts_tablet_manager.cc:1918] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:53:08.840746 15060 log.cc:1196] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/14781f4967c64354bc5aae8af4077e06
I20260620 21:53:08.841073 15060 ts_tablet_manager.cc:1939] T 14781f4967c64354bc5aae8af4077e06 P eabe2a80359f42fab4c94dba012ad0bd: Deleting consensus metadata
I20260620 21:53:08.842118 15058 ts_tablet_manager.cc:1918] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:53:08.842331 15058 log.cc:1196] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/14781f4967c64354bc5aae8af4077e06
I20260620 21:53:08.842734 15058 ts_tablet_manager.cc:1939] T 14781f4967c64354bc5aae8af4077e06 P fa52ccbd3c1045ed9caaaf7b025c686d: Deleting consensus metadata
I20260620 21:53:08.842957 14544 catalog_manager.cc:4841] TS eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413): tablet 14781f4967c64354bc5aae8af4077e06 (table test-table [id=b2bac02183c14cb791ff6be4a9a040e7]) successfully deleted
W20260620 21:53:08.843711 14543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:33441 (error 108)
I20260620 21:53:08.844075 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 14753
I20260620 21:53:08.845660 14544 catalog_manager.cc:4841] TS fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194:40759): tablet 14781f4967c64354bc5aae8af4077e06 (table test-table [id=b2bac02183c14cb791ff6be4a9a040e7]) successfully deleted
I20260620 21:53:08.846350 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:08.846051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:33441 (local address 127.0.0.1:48488)
0620 21:53:08.846205 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:08.846272 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:33441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 21:53:08.846616 14543 catalog_manager.cc:4584] TS e19037361c4a449f88e644722ff65f3b (127.0.193.193:33441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14781f4967c64354bc5aae8af4077e06: Network error: Client connection negotiation failed: client connection to 127.0.193.193:33441: connect: Connection refused (error 111)
I20260620 21:53:08.850980 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 14887
I20260620 21:53:08.856923 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 14525
I20260620 21:53:08.864065 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:35605
--webserver_interface=127.0.193.254
--webserver_port=41817
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:35605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:53:08.979972 15062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:08.980300 15062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:08.980404 15062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:08.984689 15062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:53:08.984793 15062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:08.984833 15062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:08.984879 15062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:53:08.984910 15062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:53:08.990475 15062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:35605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:35605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=41817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:08.992240 15062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:08.993377 15062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:09.007841 15069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:09.007838 15068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:09.007831 15071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:09.008298 15062 server_base.cc:1034] running on GCE node
I20260620 21:53:09.008998 15062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:09.010304 15062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:09.011497 15062 hybrid_clock.cc:648] HybridClock initialized: now 1781992389011470 us; error 42 us; skew 500 ppm
Jun 20 21:53:09 dist-test-slave-kk5c krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:09.014442 15062 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:53:09.015501 15062 webserver.cc:460] Webserver started at http://127.0.193.254:41817/ using document root <none> and password file <none>
I20260620 21:53:09.016067 15062 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:09.016145 15062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:09.020591 15062 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:53:09.023000 15078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:09.024524 15062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:53:09.024684 15062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "24805f458774465298e06cac6c76bfbb"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "92a1e4c223c725e4c66100ea1219fa19"
server_key_iv: "e616f0ccb13c03cbd3a491c38655cd47"
server_key_version: "encryptionkey@0"
I20260620 21:53:09.025393 15062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:09.042639 15062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:09.045956 15062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:09.046177 15062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:09.053779 15062 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:35605
I20260620 21:53:09.054109 15130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:35605 every 8 connection(s)
I20260620 21:53:09.054956 15062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:53:09.059638 15131 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:53:09.061370 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 15062
I20260620 21:53:09.061467 15131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Bootstrap starting.
I20260620 21:53:09.061842 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:33441
--local_ip_for_outbound_sockets=127.0.193.193
--tserver_master_addrs=127.0.193.254:35605
--webserver_port=41259
--webserver_interface=127.0.193.193
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:53:09.079728 15131 log.cc:824] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:09.089316 15131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:53:09.089769 15131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Bootstrap complete.
I20260620 21:53:09.094113 15131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } }
I20260620 21:53:09.094604 15131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24805f458774465298e06cac6c76bfbb, State: Initialized, Role: FOLLOWER
I20260620 21:53:09.095136 15131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } }
I20260620 21:53:09.095316 15131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:53:09.095427 15131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:53:09.095552 15131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:53:09.097752 15131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } }
I20260620 21:53:09.098135 15131 leader_election.cc:302] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24805f458774465298e06cac6c76bfbb; no voters:
I20260620 21:53:09.098539 15131 leader_election.cc:288] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 21:53:09.098634 15135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:53:09.098866 15135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 2 LEADER]: Becoming Leader. State: Replica: 24805f458774465298e06cac6c76bfbb, State: Running, Role: LEADER
I20260620 21:53:09.099229 15135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } }
I20260620 21:53:09.099920 15131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:53:09.100243 15137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24805f458774465298e06cac6c76bfbb. Latest consensus state: current_term: 2 leader_uuid: "24805f458774465298e06cac6c76bfbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } } }
I20260620 21:53:09.100381 15137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: This master's current role is: LEADER
I20260620 21:53:09.101058 15136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "24805f458774465298e06cac6c76bfbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } } }
I20260620 21:53:09.101155 15136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: This master's current role is: LEADER
I20260620 21:53:09.105441 15151 catalog_manager.cc:1259] Loaded cluster ID: fe7cda4d4e50402cbe0db6b3a4965fbc
I20260620 21:53:09.105526 15151 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: loading cluster ID for follower catalog manager: success
W20260620 21:53:09.106539 15151 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260620 21:53:09.106618 15151 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260620 21:53:09.106709 15139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:53:09.109464 15139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:53:09.110106 15139 catalog_manager.cc:1259] Loaded cluster ID: fe7cda4d4e50402cbe0db6b3a4965fbc
I20260620 21:53:09.110172 15139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:53:09.111904 15139 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 21:53:09.117362 15139 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 21:53:09.123781 15139 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:53:09.124730 15139 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Loaded TSK: 0
I20260620 21:53:09.126039 15139 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:53:09.190397 15133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:09.190722 15133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:09.190804 15133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:09.195130 15133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:09.195273 15133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:09.195374 15133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:53:09.201212 15133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:33441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=41259
--tserver_master_addrs=127.0.193.254:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:09.202443 15133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:09.203641 15133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:09.214044 15158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:09.214071 15159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:09.214251 15133 server_base.cc:1034] running on GCE node
W20260620 21:53:09.214071 15161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:09.214869 15133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:09.215643 15133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:09.216858 15133 hybrid_clock.cc:648] HybridClock initialized: now 1781992389216826 us; error 52 us; skew 500 ppm
Jun 20 21:53:09 dist-test-slave-kk5c krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:09.219900 15133 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:53:09.220947 15133 webserver.cc:460] Webserver started at http://127.0.193.193:41259/ using document root <none> and password file <none>
I20260620 21:53:09.221554 15133 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:09.221632 15133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:09.225862 15133 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:53:09.228585 15168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:09.230275 15133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:09.230451 15133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "e19037361c4a449f88e644722ff65f3b"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "fd5aa02cdb02a9c9bb30b7b27d2b96aa"
server_key_iv: "bab7344b5aa1194dc4f38dbe22276e4d"
server_key_version: "encryptionkey@0"
I20260620 21:53:09.231341 15133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:09.249955 15133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:09.253515 15133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:09.253731 15133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:09.254354 15133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:09.255621 15175 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 21:53:09.260214 15133 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 21:53:09.260317 15133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 21:53:09.260421 15133 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 21:53:09.262197 15133 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 21:53:09.262276 15133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 21:53:09.262598 15175 tablet_bootstrap.cc:492] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Bootstrap starting.
I20260620 21:53:09.273589 15133 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:33441
I20260620 21:53:09.273685 15282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:33441 every 8 connection(s)
I20260620 21:53:09.274646 15133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:53:09.279623 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 15133
I20260620 21:53:09.280154 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:40759
--local_ip_for_outbound_sockets=127.0.193.194
--tserver_master_addrs=127.0.193.254:35605
--webserver_port=45283
--webserver_interface=127.0.193.194
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:53:09.283869 15175 log.cc:824] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:53:09 dist-test-slave-kk5c krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:09.292299 15175 tablet_bootstrap.cc:492] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:53:09.292770 15175 tablet_bootstrap.cc:492] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Bootstrap complete.
I20260620 21:53:09.294068 15175 ts_tablet_manager.cc:1397] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.004s
I20260620 21:53:09.295975 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.276949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:49409 (local address 127.0.193.254:35605)
0620 21:53:09.278716 (+ 1767us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:09.278731 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:09.278765 (+ 34us) server_negotiation.cc:407] Connection header received
0620 21:53:09.279001 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:09.279011 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:09.279471 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:09.280615 (+ 1144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:09.281561 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.283106 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.285109 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.285813 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.285996 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.289448 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:09.289459 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:09.289479 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:09.289527 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:09.292127 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:09.292896 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:09.292903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:09.292909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:09.293071 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:09.293503 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:09.293506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:09.293508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:09.293970 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:09.294093 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:09.294772 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 21:53:09.295093 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1630,"thread_start_us":164,"threads_started":1}
I20260620 21:53:09.296000 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.276854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:49409)
0620 21:53:09.277395 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:09.277453 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:09.278309 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:09.279978 (+ 1669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:09.279988 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:09.280565 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:09.281325 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.281354 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.283343 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.283350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.284938 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.284955 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.285999 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.286006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.286260 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.287065 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:09.287092 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:09.289163 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:09.292295 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:09.292301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:09.292324 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:09.292709 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:09.293195 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:09.293201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:09.293206 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:09.293399 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:09.294111 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:09.294120 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:09.294424 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 21:53:09.294630 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 21:53:09.294977 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":179,"threads_started":1}
I20260620 21:53:09.297255 15175 raft_consensus.cc:348] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } } peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } }
I20260620 21:53:09.297785 15175 raft_consensus.cc:729] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19037361c4a449f88e644722ff65f3b, State: Initialized, Role: FOLLOWER
I20260620 21:53:09.298346 15175 consensus_queue.cc:260] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } } peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } }
I20260620 21:53:09.299543 15175 ts_tablet_manager.cc:1428] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260620 21:53:09.299580 15283 heartbeater.cc:347] Connected to a master server at 127.0.193.254:35605
I20260620 21:53:09.299822 15283 heartbeater.cc:464] Registering TS with master...
I20260620 21:53:09.300400 15283 heartbeater.cc:511] Master 127.0.193.254:35605 requested a full tablet report, sending...
I20260620 21:53:09.303092 15095 ts_manager.cc:194] Registered new tserver with Master: e19037361c4a449f88e644722ff65f3b (127.0.193.193:33441)
I20260620 21:53:09.306754 15095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:49409
I20260620 21:53:09.309473 15283 heartbeater.cc:503] Master 127.0.193.254:35605 was elected leader, sending a full tablet report...
Jun 20 21:53:09 dist-test-slave-kk5c krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for kudu/127.0.193.193@KRBTEST.COM
I20260620 21:53:09.317456 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.306187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:33441 (local address 127.0.0.1:48494)
0620 21:53:09.306951 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:09.306994 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:09.307135 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:09.307692 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:09.307698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:09.308112 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:09.308222 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.308228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.309215 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.309220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.310001 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.310014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.310771 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.310774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.310884 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.311605 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:09.311617 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:09.313407 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:09.315479 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:09.315488 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:09.315492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:09.315758 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:09.316085 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:09.316088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:09.316089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:09.316172 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:09.316708 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:09.316712 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:09.316820 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 21:53:09.317112 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 21:53:09.317302 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":131,"threads_started":1}
I20260620 21:53:09.317550 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.306285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48494 (local address 127.0.193.193:33441)
0620 21:53:09.306681 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:09.306692 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:09.307104 (+ 412us) server_negotiation.cc:407] Connection header received
0620 21:53:09.307305 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:09.307310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:09.307532 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:09.307667 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:09.308369 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.309087 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.310122 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.310625 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.310736 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.313557 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:09.313568 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:09.313581 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:09.313610 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:09.315360 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:09.315875 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:09.315881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:09.315883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:09.315976 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:09.316274 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:09.316277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:09.316279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:09.316595 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:09.316724 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:09.317199 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 21:53:09.317388 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":98,"threads_started":1}
I20260620 21:53:09.318783 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet 14781f4967c64354bc5aae8af4077e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:08 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:48494
I20260620 21:53:09.319837 15292 tablet_replica.cc:330] stopping tablet replica
I20260620 21:53:09.320175 15292 raft_consensus.cc:2229] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:53:09.320452 15292 raft_consensus.cc:2258] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:53:09.321666 15292 ts_tablet_manager.cc:1905] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:53:09.326584 15292 ts_tablet_manager.cc:1918] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:53:09.326699 15292 log.cc:1196] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal/wals/14781f4967c64354bc5aae8af4077e06
I20260620 21:53:09.326972 15292 ts_tablet_manager.cc:1939] T 14781f4967c64354bc5aae8af4077e06 P e19037361c4a449f88e644722ff65f3b: Deleting consensus metadata
I20260620 21:53:09.329116 15081 catalog_manager.cc:4841] TS e19037361c4a449f88e644722ff65f3b (127.0.193.193:33441): tablet 14781f4967c64354bc5aae8af4077e06 (table test-table [id=b2bac02183c14cb791ff6be4a9a040e7]) successfully deleted
W20260620 21:53:09.406778 15287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:09.407042 15287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:09.407086 15287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:09.411156 15287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:09.411298 15287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:09.411375 15287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:53:09.416786 15287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:40759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=45283
--tserver_master_addrs=127.0.193.254:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:09.417881 15287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:09.418841 15287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:09.428634 15300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:09.428627 15298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:09.428627 15297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:09.428785 15287 server_base.cc:1034] running on GCE node
I20260620 21:53:09.429405 15287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:09.430231 15287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:09.431394 15287 hybrid_clock.cc:648] HybridClock initialized: now 1781992389431376 us; error 35 us; skew 500 ppm
Jun 20 21:53:09 dist-test-slave-kk5c krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:09.434139 15287 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:53:09.435056 15287 webserver.cc:460] Webserver started at http://127.0.193.194:45283/ using document root <none> and password file <none>
I20260620 21:53:09.435709 15287 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:09.435786 15287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:09.439953 15287 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:09.442070 15307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:09.443531 15287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:09.443655 15287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "fa52ccbd3c1045ed9caaaf7b025c686d"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "9e95dd4cd283d2ee21d477c8fa568744"
server_key_iv: "b23038021fa6d09c7b8e2f9ea62f9f1e"
server_key_version: "encryptionkey@0"
I20260620 21:53:09.444207 15287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:09.465567 15287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:09.469039 15287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:09.469270 15287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:09.469945 15287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:09.470818 15287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:53:09.470875 15287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:09.470950 15287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:53:09.470984 15287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:09.480813 15287 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:40759
I20260620 21:53:09.480830 15420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:40759 every 8 connection(s)
I20260620 21:53:09.481972 15287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:53:09.488565 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 15287
I20260620 21:53:09.489092 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:37413
--local_ip_for_outbound_sockets=127.0.193.195
--tserver_master_addrs=127.0.193.254:35605
--webserver_port=36685
--webserver_interface=127.0.193.195
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:53:09 dist-test-slave-kk5c krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:09.497710 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.484160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:53411 (local address 127.0.193.254:35605)
0620 21:53:09.484328 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:09.484333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:09.485569 (+ 1236us) server_negotiation.cc:407] Connection header received
0620 21:53:09.486485 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:09.486488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:09.486542 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:09.486628 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:09.488069 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.488917 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.490325 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.490575 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.490700 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.493619 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:09.493623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:09.493625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:09.493650 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:09.495489 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:09.496072 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:09.496076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:09.496078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:09.496161 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:09.496544 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:09.496549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:09.496551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:09.496749 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:09.496857 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:09.497391 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 21:53:09.497533 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:53:09.498843 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.484640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:53411)
0620 21:53:09.485401 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:09.485447 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:09.486252 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:09.486863 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:09.486870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:09.487361 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:09.487879 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.487896 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.489104 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.489107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.490175 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.490186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.490749 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.490753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.490935 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.491625 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:09.491650 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:09.493399 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:09.495623 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:09.495629 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:09.495649 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:09.495950 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:09.496285 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:09.496288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:09.496290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:09.496442 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:09.496857 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:09.496865 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:09.497128 (+ 263us) client_negotiation.cc:769] Sending connection context
0620 21:53:09.497326 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 21:53:09.497679 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":127,"threads_started":1}
I20260620 21:53:09.500515 15421 heartbeater.cc:347] Connected to a master server at 127.0.193.254:35605
I20260620 21:53:09.500773 15421 heartbeater.cc:464] Registering TS with master...
I20260620 21:53:09.501382 15421 heartbeater.cc:511] Master 127.0.193.254:35605 requested a full tablet report, sending...
I20260620 21:53:09.502559 15095 ts_manager.cc:194] Registered new tserver with Master: fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194:40759)
I20260620 21:53:09.503364 15095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:53411
W20260620 21:53:09.611048 15424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:09.611500 15424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:09.611588 15424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:09.616459 15424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:09.616561 15424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:09.616662 15424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:53:09.622453 15424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:37413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=36685
--tserver_master_addrs=127.0.193.254:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:09.623728 15424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:09.624907 15424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:09.635025 15430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:09.635210 15429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:09.635401 15432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:09.635581 15424 server_base.cc:1034] running on GCE node
I20260620 21:53:09.635993 15424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:09.636873 15424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:09.638060 15424 hybrid_clock.cc:648] HybridClock initialized: now 1781992389638043 us; error 30 us; skew 500 ppm
Jun 20 21:53:09 dist-test-slave-kk5c krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:09.641237 15424 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:53:09.642371 15424 webserver.cc:460] Webserver started at http://127.0.193.195:36685/ using document root <none> and password file <none>
I20260620 21:53:09.643030 15424 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:09.643160 15424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:09.647536 15424 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 21:53:09.649983 15439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:09.651443 15424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:09.651595 15424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "eabe2a80359f42fab4c94dba012ad0bd"
format_stamp: "Formatted at 2026-06-20 21:53:08 on dist-test-slave-kk5c"
server_key: "a0c23a279ed342e1954c66e097556ada"
server_key_iv: "f4e759157779b639e3c0d091ded09691"
server_key_version: "encryptionkey@0"
I20260620 21:53:09.652197 15424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:09.663609 15424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:09.667026 15424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:09.667312 15424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:09.668004 15424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:09.668902 15424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:53:09.668982 15424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:09.669061 15424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:53:09.669109 15424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:09.679234 15424 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:37413
I20260620 21:53:09.679351 15552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:37413 every 8 connection(s)
I20260620 21:53:09.680368 15424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:53:09.686666 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 15424
Jun 20 21:53:09 dist-test-slave-kk5c krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:09.694711 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.682267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:33609 (local address 127.0.193.254:35605)
0620 21:53:09.682429 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:09.682433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:09.683645 (+ 1212us) server_negotiation.cc:407] Connection header received
0620 21:53:09.684569 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:09.684573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:09.684635 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:09.684744 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:09.686003 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.686559 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.688054 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.688278 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.688402 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.690898 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:09.690902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:09.690904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:09.690928 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:09.692508 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:09.693093 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:09.693097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:09.693099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:09.693183 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:09.693549 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:09.693553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:09.693555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:09.693744 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:09.693859 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:09.694420 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 21:53:09.694543 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 21:53:09.695859 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.682694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:33609)
0620 21:53:09.683487 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:09.683533 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:09.684360 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:09.684931 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:09.684939 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:09.685310 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:09.685836 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.685853 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.686696 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.686704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.687921 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.687932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.688448 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.688451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.688625 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.689249 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:09.689266 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:09.690707 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:09.692644 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:09.692650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:09.692668 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:09.692976 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:09.693281 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:09.693284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:09.693287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:09.693446 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:09.693861 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:09.693871 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:09.694167 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 21:53:09.694369 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 21:53:09.694730 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":127,"threads_started":1}
I20260620 21:53:09.696652 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.687565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44880 (local address 127.0.193.254:35605)
0620 21:53:09.687925 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:09.687930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:09.687946 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:53:09.688095 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:09.688098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:09.688147 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:09.688262 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:09.689055 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.689623 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.690666 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.690943 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.691067 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.692540 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:09.692544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:09.692547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:09.692579 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:09.694245 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:09.694770 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:09.694773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:09.694775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:09.694895 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:09.695252 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:09.695256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:09.695258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:09.695475 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:09.695632 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:09.695888 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 21:53:09.696524 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":88,"threads_started":1}
I20260620 21:53:09.697198 15553 heartbeater.cc:347] Connected to a master server at 127.0.193.254:35605
I20260620 21:53:09.697466 15553 heartbeater.cc:464] Registering TS with master...
I20260620 21:53:09.698038 15553 heartbeater.cc:511] Master 127.0.193.254:35605 requested a full tablet report, sending...
I20260620 21:53:09.699071 15095 ts_manager.cc:194] Registered new tserver with Master: eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413)
I20260620 21:53:09.699713 15095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:33609
I20260620 21:53:09.710867 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:53:09.716301 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.711839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.0.193.254:35605)
0620 21:53:09.712057 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:09.712061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:09.712213 (+ 152us) server_negotiation.cc:407] Connection header received
0620 21:53:09.712427 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:09.712430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:09.712547 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:09.712649 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:53:09.712908 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.713537 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.714519 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.714807 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.714950 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.715378 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:53:09.715846 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:53:09.715950 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:09.716051 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 21:53:09.716117 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 21:53:09.717079 15095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:53:09.720691 15095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:53:09.729300 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 26921abdd88748689b2bc89402d55cb0 (DEFAULT_TABLE table=test-table [id=e2e55aba54134cd1886df8ea19f75b4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:53:09.730008 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26921abdd88748689b2bc89402d55cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:09.733170 15175 tablet_bootstrap.cc:492] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b: Bootstrap starting.
I20260620 21:53:09.733680 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.728135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:37413 (local address 127.0.0.1:50000)
0620 21:53:09.728302 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:09.728317 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:09.728401 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:09.729250 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:09.729253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:09.729273 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:09.729410 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.729418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.730734 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.730737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.732016 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.732031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.733222 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.733229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.733352 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.733414 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 21:53:09.733468 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 21:53:09.733557 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 21:53:09.734125 15175 tablet_bootstrap.cc:654] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:09.734515 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.728291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.0.193.195:37413)
0620 21:53:09.728685 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:09.728698 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:09.728721 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:53:09.728849 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:09.728855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:09.729076 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:09.729269 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:09.729591 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.730592 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.732184 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.733074 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.733997 (+ 923us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.734145 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:09.734226 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 21:53:09.734350 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":170,"threads_started":1}
I20260620 21:53:09.734803 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.729101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.0.193.194:40759)
0620 21:53:09.729713 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:09.729727 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:09.729749 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:53:09.729821 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:09.729826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:09.730089 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:09.730242 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:09.730695 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.731613 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.733154 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.734096 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.734219 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.734373 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:09.734509 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 21:53:09.734636 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":105,"threads_started":1}
I20260620 21:53:09.735005 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.728623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:40759 (local address 127.0.0.1:34944)
0620 21:53:09.729597 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:09.729611 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:09.729702 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:09.730427 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:09.730429 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:09.730448 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:09.730546 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.730556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.731758 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.731762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.732993 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.733003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.734279 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.734283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.734375 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.734399 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:53:09.734479 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 21:53:09.734727 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":111,"threads_started":1}
I20260620 21:53:09.735793 15175 tablet_bootstrap.cc:492] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b: No bootstrap required, opened a new log
I20260620 21:53:09.735891 15175 ts_tablet_manager.cc:1397] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 21:53:09.736404 15175 raft_consensus.cc:348] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:09.736536 15175 raft_consensus.cc:374] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:09.736578 15175 raft_consensus.cc:729] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19037361c4a449f88e644722ff65f3b, State: Initialized, Role: FOLLOWER
I20260620 21:53:09.736688 15175 consensus_queue.cc:260] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:09.736933 15175 ts_tablet_manager.cc:1428] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 21:53:09.737819 15487 tablet_service.cc:1467] Processing CreateTablet for tablet 26921abdd88748689b2bc89402d55cb0 (DEFAULT_TABLE table=test-table [id=e2e55aba54134cd1886df8ea19f75b4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:53:09.738127 15355 tablet_service.cc:1467] Processing CreateTablet for tablet 26921abdd88748689b2bc89402d55cb0 (DEFAULT_TABLE table=test-table [id=e2e55aba54134cd1886df8ea19f75b4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:53:09.738863 15487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26921abdd88748689b2bc89402d55cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:09.739141 15355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26921abdd88748689b2bc89402d55cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:09.745724 15566 tablet_bootstrap.cc:492] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd: Bootstrap starting.
I20260620 21:53:09.745816 15567 tablet_bootstrap.cc:492] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d: Bootstrap starting.
I20260620 21:53:09.748445 15566 tablet_bootstrap.cc:654] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:09.748540 15567 tablet_bootstrap.cc:654] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:09.749321 15567 log.cc:824] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:09.749356 15566 log.cc:824] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:09.751468 15567 tablet_bootstrap.cc:492] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d: No bootstrap required, opened a new log
I20260620 21:53:09.751541 15566 tablet_bootstrap.cc:492] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd: No bootstrap required, opened a new log
I20260620 21:53:09.751680 15567 ts_tablet_manager.cc:1397] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:53:09.751716 15566 ts_tablet_manager.cc:1397] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:53:09.754592 15566 raft_consensus.cc:348] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:09.754592 15567 raft_consensus.cc:348] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:09.754889 15566 raft_consensus.cc:374] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:09.754882 15567 raft_consensus.cc:374] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:09.754951 15566 raft_consensus.cc:729] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eabe2a80359f42fab4c94dba012ad0bd, State: Initialized, Role: FOLLOWER
I20260620 21:53:09.754989 15567 raft_consensus.cc:729] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa52ccbd3c1045ed9caaaf7b025c686d, State: Initialized, Role: FOLLOWER
I20260620 21:53:09.755498 15566 consensus_queue.cc:260] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:09.755517 15567 consensus_queue.cc:260] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:09.756572 15553 heartbeater.cc:503] Master 127.0.193.254:35605 was elected leader, sending a full tablet report...
I20260620 21:53:09.756640 15421 heartbeater.cc:503] Master 127.0.193.254:35605 was elected leader, sending a full tablet report...
I20260620 21:53:09.756832 15567 ts_tablet_manager.cc:1428] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260620 21:53:09.757682 15566 ts_tablet_manager.cc:1428] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 21:53:09.776053 15284 tablet.cc:2369] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:53:09.803741 15570 raft_consensus.cc:482] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:53:09.803978 15570 raft_consensus.cc:504] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:09.805238 15570 leader_election.cc:288] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413), e19037361c4a449f88e644722ff65f3b (127.0.193.193:33441)
I20260620 21:53:09.809188 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.805515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:37413 (local address 127.0.193.194:40531)
0620 21:53:09.806009 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:09.806020 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:09.806156 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:09.806404 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:09.806407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:09.806423 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:09.806540 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.806545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.807261 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.807268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.808089 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.808097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.808746 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.808750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.808844 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.808871 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 21:53:09.808964 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 21:53:09.809037 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":129,"thread_start_us":113,"threads_started":1}
I20260620 21:53:09.809172 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.805691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:40531 (local address 127.0.193.195:37413)
0620 21:53:09.805821 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:09.805825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:09.806112 (+ 287us) server_negotiation.cc:407] Connection header received
0620 21:53:09.806266 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:09.806269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:09.806310 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:09.806402 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:09.806646 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.807128 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.808215 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.808622 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.808752 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.808807 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:09.808977 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 21:53:09.809041 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:53:09.809340 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.805515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:33441 (local address 127.0.193.194:38571)
0620 21:53:09.805699 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:09.805719 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:09.805829 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:09.806180 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:09.806183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:09.806201 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:09.806332 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.806338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.807155 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.807162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.808113 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:09.808123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.808887 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.808891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:09.809010 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.809032 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 21:53:09.809129 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 21:53:09.809205 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":50}
I20260620 21:53:09.809340 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.805660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:38571 (local address 127.0.193.193:33441)
0620 21:53:09.805799 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:09.805804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:09.805820 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:53:09.805979 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:09.805982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:09.806046 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:09.806155 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:09.806467 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.807035 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.808252 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.808744 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.808892 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.809053 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:09.809142 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 21:53:09.809218 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:53:09.809947 15507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26921abdd88748689b2bc89402d55cb0" candidate_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eabe2a80359f42fab4c94dba012ad0bd" is_pre_election: true
I20260620 21:53:09.809971 15236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26921abdd88748689b2bc89402d55cb0" candidate_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19037361c4a449f88e644722ff65f3b" is_pre_election: true
I20260620 21:53:09.810250 15507 raft_consensus.cc:2454] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa52ccbd3c1045ed9caaaf7b025c686d in term 0.
I20260620 21:53:09.810268 15236 raft_consensus.cc:2454] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa52ccbd3c1045ed9caaaf7b025c686d in term 0.
I20260620 21:53:09.810717 15311 leader_election.cc:302] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: e19037361c4a449f88e644722ff65f3b, eabe2a80359f42fab4c94dba012ad0bd, fa52ccbd3c1045ed9caaaf7b025c686d; no voters:
I20260620 21:53:09.811038 15570 raft_consensus.cc:2781] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:53:09.811120 15570 raft_consensus.cc:482] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:53:09.811177 15570 raft_consensus.cc:3037] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:09.812364 15570 raft_consensus.cc:504] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:09.812801 15570 leader_election.cc:288] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [CANDIDATE]: Term 1 election: Requested vote from peers eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413), e19037361c4a449f88e644722ff65f3b (127.0.193.193:33441)
I20260620 21:53:09.813198 15236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26921abdd88748689b2bc89402d55cb0" candidate_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19037361c4a449f88e644722ff65f3b"
I20260620 21:53:09.813208 15507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26921abdd88748689b2bc89402d55cb0" candidate_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eabe2a80359f42fab4c94dba012ad0bd"
I20260620 21:53:09.813351 15236 raft_consensus.cc:3037] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:09.813367 15507 raft_consensus.cc:3037] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:09.814388 15236 raft_consensus.cc:2454] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa52ccbd3c1045ed9caaaf7b025c686d in term 1.
I20260620 21:53:09.814575 15507 raft_consensus.cc:2454] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa52ccbd3c1045ed9caaaf7b025c686d in term 1.
I20260620 21:53:09.814774 15310 leader_election.cc:302] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e19037361c4a449f88e644722ff65f3b, fa52ccbd3c1045ed9caaaf7b025c686d; no voters:
I20260620 21:53:09.815078 15570 raft_consensus.cc:2781] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:53:09.815425 15570 raft_consensus.cc:686] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [term 1 LEADER]: Becoming Leader. State: Replica: fa52ccbd3c1045ed9caaaf7b025c686d, State: Running, Role: LEADER
I20260620 21:53:09.815815 15570 consensus_queue.cc:237] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:09.819319 15095 catalog_manager.cc:5495] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d reported cstate change: term changed from 0 to 1, leader changed from <none> to fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194). New cstate: current_term: 1 leader_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } health_report { overall_health: UNKNOWN } } }
I20260620 21:53:09.846709 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.842696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.0.193.194:40759)
0620 21:53:09.842846 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:09.842850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:09.842871 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:53:09.843027 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:09.843029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:09.843089 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:09.843238 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:53:09.843457 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.844003 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.844913 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:09.845153 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:09.845242 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:09.845743 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:53:09.846215 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:53:09.846330 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:09.846492 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 21:53:09.846558 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 21:53:09.853736 15236 raft_consensus.cc:1261] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Refusing update from remote peer fa52ccbd3c1045ed9caaaf7b025c686d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:53:09.853739 15507 raft_consensus.cc:1261] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd [term 1 FOLLOWER]: Refusing update from remote peer fa52ccbd3c1045ed9caaaf7b025c686d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:53:09.854571 15573 consensus_queue.cc:1035] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:53:09.854779 15570 consensus_queue.cc:1035] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:53:09.870637 15093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:53:09.870859 15093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:53:09.873525 15093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Sending DeleteTablet for 3 replicas of tablet 26921abdd88748689b2bc89402d55cb0
I20260620 21:53:09.874300 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet 26921abdd88748689b2bc89402d55cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:09 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:48494
I20260620 21:53:09.874383 15355 tablet_service.cc:1514] Processing DeleteTablet for tablet 26921abdd88748689b2bc89402d55cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:09 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:34944
I20260620 21:53:09.874418 15487 tablet_service.cc:1514] Processing DeleteTablet for tablet 26921abdd88748689b2bc89402d55cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:09 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:50000
I20260620 21:53:09.874887 15586 tablet_replica.cc:330] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d: stopping tablet replica
I20260620 21:53:09.875030 15587 tablet_replica.cc:330] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd: stopping tablet replica
I20260620 21:53:09.875068 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 15133
I20260620 21:53:09.875695 15586 raft_consensus.cc:2229] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:53:09.876125 15586 raft_consensus.cc:2258] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:53:09.876773 15587 raft_consensus.cc:2229] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:53:09.877223 15587 raft_consensus.cc:2258] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:53:09.877972 15586 ts_tablet_manager.cc:1905] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:53:09.879058 15587 ts_tablet_manager.cc:1905] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:53:09.881685 15586 ts_tablet_manager.cc:1918] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:53:09.881829 15586 log.cc:1196] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/26921abdd88748689b2bc89402d55cb0
I20260620 21:53:09.882189 15586 ts_tablet_manager.cc:1939] T 26921abdd88748689b2bc89402d55cb0 P fa52ccbd3c1045ed9caaaf7b025c686d: Deleting consensus metadata
I20260620 21:53:09.882334 15587 ts_tablet_manager.cc:1918] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:53:09.882513 15587 log.cc:1196] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/26921abdd88748689b2bc89402d55cb0
I20260620 21:53:09.882946 15587 ts_tablet_manager.cc:1939] T 26921abdd88748689b2bc89402d55cb0 P eabe2a80359f42fab4c94dba012ad0bd: Deleting consensus metadata
I20260620 21:53:09.883865 15082 catalog_manager.cc:4841] TS fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194:40759): tablet 26921abdd88748689b2bc89402d55cb0 (table test-table [id=e2e55aba54134cd1886df8ea19f75b4a]) successfully deleted
I20260620 21:53:09.884238 15082 catalog_manager.cc:4841] TS eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413): tablet 26921abdd88748689b2bc89402d55cb0 (table test-table [id=e2e55aba54134cd1886df8ea19f75b4a]) successfully deleted
W20260620 21:53:09.884253 15081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:33441 (error 108)
I20260620 21:53:09.884754 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 15287
I20260620 21:53:09.886876 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:09.886494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:33441 (local address 127.0.0.1:48508)
0620 21:53:09.886674 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:09.886765 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:33441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 21:53:09.887089 15081 catalog_manager.cc:4584] TS e19037361c4a449f88e644722ff65f3b (127.0.193.193:33441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26921abdd88748689b2bc89402d55cb0: Network error: Client connection negotiation failed: client connection to 127.0.193.193:33441: connect: Connection refused (error 111)
I20260620 21:53:09.891364 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 15424
I20260620 21:53:09.897395 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 15062
I20260620 21:53:09.904309 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:35605
--webserver_interface=127.0.193.254
--webserver_port=41817
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:35605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:53:10.021114 15589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:10.021415 15589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:10.021458 15589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:10.025709 15589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:53:10.025776 15589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:10.025797 15589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:10.025815 15589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:53:10.025830 15589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:53:10.031704 15589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:35605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:35605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=41817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:10.032876 15589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:10.033951 15589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:53:10.043135 15589 server_base.cc:1034] running on GCE node
W20260620 21:53:10.043068 15594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:10.043067 15595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:10.043067 15597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:10.044101 15589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:10.045274 15589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:10.046502 15589 hybrid_clock.cc:648] HybridClock initialized: now 1781992390046442 us; error 48 us; skew 500 ppm
Jun 20 21:53:10 dist-test-slave-kk5c krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:10.049542 15589 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:53:10.050542 15589 webserver.cc:460] Webserver started at http://127.0.193.254:41817/ using document root <none> and password file <none>
I20260620 21:53:10.051133 15589 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:10.051265 15589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:10.055154 15589 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:53:10.057565 15604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:10.059334 15589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:10.059456 15589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "24805f458774465298e06cac6c76bfbb"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "92a1e4c223c725e4c66100ea1219fa19"
server_key_iv: "e616f0ccb13c03cbd3a491c38655cd47"
server_key_version: "encryptionkey@0"
I20260620 21:53:10.060072 15589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:10.078155 15589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:10.081828 15589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:10.082005 15589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:10.089658 15589 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:35605
I20260620 21:53:10.089970 15656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:35605 every 8 connection(s)
I20260620 21:53:10.090857 15589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:53:10.091722 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 15589
I20260620 21:53:10.092202 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:33441
--local_ip_for_outbound_sockets=127.0.193.193
--tserver_master_addrs=127.0.193.254:35605
--webserver_port=41259
--webserver_interface=127.0.193.193
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:53:10.096848 15657 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:53:10.100070 15657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Bootstrap starting.
I20260620 21:53:10.123879 15657 log.cc:824] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:10.139580 15657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:53:10.140060 15657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb: Bootstrap complete.
I20260620 21:53:10.144218 15657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } }
I20260620 21:53:10.144644 15657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24805f458774465298e06cac6c76bfbb, State: Initialized, Role: FOLLOWER
I20260620 21:53:10.145148 15657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } }
I20260620 21:53:10.145303 15657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:53:10.145395 15657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:53:10.145526 15657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 2 FOLLOWER]: Advancing to term 3
I20260620 21:53:10.147644 15657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } }
I20260620 21:53:10.148008 15657 leader_election.cc:302] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24805f458774465298e06cac6c76bfbb; no voters:
I20260620 21:53:10.148373 15657 leader_election.cc:288] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 21:53:10.148516 15661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 3 FOLLOWER]: Leader election won for term 3
I20260620 21:53:10.148775 15661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [term 3 LEADER]: Becoming Leader. State: Replica: 24805f458774465298e06cac6c76bfbb, State: Running, Role: LEADER
I20260620 21:53:10.149096 15661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } }
I20260620 21:53:10.149629 15657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:53:10.150154 15661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24805f458774465298e06cac6c76bfbb. Latest consensus state: current_term: 3 leader_uuid: "24805f458774465298e06cac6c76bfbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } } }
I20260620 21:53:10.150275 15661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: This master's current role is: LEADER
I20260620 21:53:10.150815 15666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:53:10.150772 15662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "24805f458774465298e06cac6c76bfbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24805f458774465298e06cac6c76bfbb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 35605 } } }
I20260620 21:53:10.151000 15662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24805f458774465298e06cac6c76bfbb [sys.catalog]: This master's current role is: LEADER
I20260620 21:53:10.156275 15666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:53:10.157123 15666 catalog_manager.cc:1259] Loaded cluster ID: fe7cda4d4e50402cbe0db6b3a4965fbc
I20260620 21:53:10.157186 15666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 21:53:10.158154 15666 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5b17c1c9ad google::LogMessage::Fail() at ??:0
@ 0x7f5b17c1f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f5b17c1c4de google::LogMessage::Flush() at ??:0
@ 0x7f5b17c1da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5b2255bfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f5b2255cd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5b22559579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f5b2259d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5b19e6e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5b19e6ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f5b19e703f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5b19e5f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5b1b4a36db start_thread at ??:0
@ 0x7f5b1671a61f clone at ??:0
W20260620 21:53:10.248835 15659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:10.249198 15659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:10.249277 15659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:10.253650 15659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:10.253759 15659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:10.253856 15659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:53:10.259910 15659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:33441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=41259
--tserver_master_addrs=127.0.193.254:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:10.261102 15659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:10.262148 15659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:10.272748 15698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:10.272750 15699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:10.273437 15701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:10.273386 15659 server_base.cc:1034] running on GCE node
I20260620 21:53:10.274135 15659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:10.275043 15659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:10.276219 15659 hybrid_clock.cc:648] HybridClock initialized: now 1781992390276199 us; error 57 us; skew 500 ppm
Jun 20 21:53:10 dist-test-slave-kk5c krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:10.279488 15659 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:53:10.280481 15659 webserver.cc:460] Webserver started at http://127.0.193.193:41259/ using document root <none> and password file <none>
I20260620 21:53:10.281081 15659 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:10.281176 15659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:10.285797 15659 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:53:10.288434 15708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:10.289925 15659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 21:53:10.290088 15659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "e19037361c4a449f88e644722ff65f3b"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "fd5aa02cdb02a9c9bb30b7b27d2b96aa"
server_key_iv: "bab7344b5aa1194dc4f38dbe22276e4d"
server_key_version: "encryptionkey@0"
I20260620 21:53:10.290685 15659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:10.301754 15659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:10.305335 15659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:10.305552 15659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:10.306180 15659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:10.307359 15715 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 21:53:10.313395 15659 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 21:53:10.313515 15659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 21:53:10.313613 15659 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 21:53:10.315472 15659 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 21:53:10.315554 15659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260620 21:53:10.315886 15715 tablet_bootstrap.cc:492] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b: Bootstrap starting.
I20260620 21:53:10.326774 15659 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:33441
I20260620 21:53:10.327056 15822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:33441 every 8 connection(s)
I20260620 21:53:10.327890 15659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:53:10.331059 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 15659
I20260620 21:53:10.331573 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:40759
--local_ip_for_outbound_sockets=127.0.193.194
--tserver_master_addrs=127.0.193.254:35605
--webserver_port=45283
--webserver_interface=127.0.193.194
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:53:10.339007 15715 log.cc:824] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:10.348536 15715 tablet_bootstrap.cc:492] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:53:10.348945 15715 tablet_bootstrap.cc:492] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b: Bootstrap complete.
I20260620 21:53:10.350081 15715 ts_tablet_manager.cc:1397] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.004s
I20260620 21:53:10.352732 15715 raft_consensus.cc:348] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:10.353181 15715 raft_consensus.cc:729] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19037361c4a449f88e644722ff65f3b, State: Initialized, Role: FOLLOWER
I20260620 21:53:10.353679 15715 consensus_queue.cc:260] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:10.354650 15715 ts_tablet_manager.cc:1428] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260620 21:53:10.422833 15825 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:53:10.329918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:46167)
0620 21:53:10.330416 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:10.330472 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:10.331919 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:10.422033 (+ 90114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":140,"threads_started":1}
W20260620 21:53:10.423477 15823 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:35605 (0 consecutive failures): Network error: Failed to ping master at 127.0.193.254:35605: Client connection negotiation failed: client connection to 127.0.193.254:35605: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 21:53:10.424279 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:10.423996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:51059)
0620 21:53:10.424137 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:10.424214 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 21:53:10.425022 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:10.424807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:60593)
0620 21:53:10.424916 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:10.424963 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 21:53:10.425218 15823 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:53:10.479372 15826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:10.479661 15826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:10.479708 15826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:10.484040 15826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:10.484112 15826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:10.484176 15826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:53:10.489687 15826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:40759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=45283
--tserver_master_addrs=127.0.193.254:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:10.490845 15826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:10.491909 15826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:10.504004 15833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:10.504148 15826 server_base.cc:1034] running on GCE node
W20260620 21:53:10.504103 15836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:10.504004 15834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:10.504835 15826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:10.505545 15826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:10.506767 15826 hybrid_clock.cc:648] HybridClock initialized: now 1781992390506694 us; error 96 us; skew 500 ppm
Jun 20 21:53:10 dist-test-slave-kk5c krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:10.509938 15826 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:53:10.510901 15826 webserver.cc:460] Webserver started at http://127.0.193.194:45283/ using document root <none> and password file <none>
I20260620 21:53:10.511639 15826 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:10.511759 15826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:10.516433 15826 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:53:10.519369 15843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:10.520942 15826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:53:10.521129 15826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "fa52ccbd3c1045ed9caaaf7b025c686d"
format_stamp: "Formatted at 2026-06-20 21:53:07 on dist-test-slave-kk5c"
server_key: "9e95dd4cd283d2ee21d477c8fa568744"
server_key_iv: "b23038021fa6d09c7b8e2f9ea62f9f1e"
server_key_version: "encryptionkey@0"
I20260620 21:53:10.521736 15826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:10.547899 15826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:10.551658 15826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:10.551875 15826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:10.552526 15826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:10.553416 15826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:53:10.553463 15826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:10.553503 15826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:53:10.553555 15826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:10.563037 15826 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:40759
I20260620 21:53:10.563097 15956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:40759 every 8 connection(s)
I20260620 21:53:10.564224 15826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:53:10.565083 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 15826
I20260620 21:53:10.565479 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:37413
--local_ip_for_outbound_sockets=127.0.193.195
--tserver_master_addrs=127.0.193.254:35605
--webserver_port=36685
--webserver_interface=127.0.193.195
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:53:10.568475 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:10.566601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:56969)
0620 21:53:10.567433 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:10.567570 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":192,"threads_started":1}
W20260620 21:53:10.569109 15957 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:35605 (0 consecutive failures): Network error: Failed to ping master at 127.0.193.254:35605: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
I20260620 21:53:10.569844 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:10.569603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:34659)
0620 21:53:10.569725 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:10.569787 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 21:53:10.570528 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:10.570317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:54461)
0620 21:53:10.570422 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:10.570469 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 21:53:10.570734 15957 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:53:10.684103 15959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:10.684399 15959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:10.684444 15959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:10.688617 15959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:10.688687 15959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:10.688748 15959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:53:10.694485 15959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:37413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=36685
--tserver_master_addrs=127.0.193.254:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:10.695744 15959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:10.696760 15959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:10.707692 15966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:10.707878 15959 server_base.cc:1034] running on GCE node
W20260620 21:53:10.707705 15965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:10.707938 15968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:10.708513 15959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:10.709210 15959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:10.710394 15959 hybrid_clock.cc:648] HybridClock initialized: now 1781992390710375 us; error 31 us; skew 500 ppm
Jun 20 21:53:10 dist-test-slave-kk5c krb5kdc[14512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:10.713907 15959 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:53:10.714820 15959 webserver.cc:460] Webserver started at http://127.0.193.195:36685/ using document root <none> and password file <none>
I20260620 21:53:10.715489 15959 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:10.715561 15959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:10.719714 15959 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:53:10.722033 15975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:10.723631 15959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:10.723793 15959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "eabe2a80359f42fab4c94dba012ad0bd"
format_stamp: "Formatted at 2026-06-20 21:53:08 on dist-test-slave-kk5c"
server_key: "a0c23a279ed342e1954c66e097556ada"
server_key_iv: "f4e759157779b639e3c0d091ded09691"
server_key_version: "encryptionkey@0"
I20260620 21:53:10.724417 15959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:10.749401 15959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:10.752871 15959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:10.753115 15959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:10.753677 15959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:10.754501 15959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:53:10.754549 15959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:10.754587 15959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:53:10.754602 15959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:10.764612 15959 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:37413
I20260620 21:53:10.764642 16088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:37413 every 8 connection(s)
I20260620 21:53:10.765867 15959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:53:10.769713 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:10.768140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:54655)
0620 21:53:10.768906 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:10.769031 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":123,"threads_started":1}
W20260620 21:53:10.770308 16089 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:35605 (0 consecutive failures): Network error: Failed to ping master at 127.0.193.254:35605: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
I20260620 21:53:10.770982 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:10.770702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:33347)
0620 21:53:10.770853 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:10.770923 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 21:53:10.771706 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:10.771492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:40467)
0620 21:53:10.771597 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:10.771644 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 21:53:10.771956 16089 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 21:53:10.773475 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 15959
I20260620 21:53:11.426380 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:11.425875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:59619)
0620 21:53:11.426130 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:11.426192 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260620 21:53:11.571875 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:11.571444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:50555)
0620 21:53:11.571703 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:11.571774 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":68,"threads_started":1}
I20260620 21:53:11.773516 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:11.772947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:39499)
0620 21:53:11.773315 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:11.773383 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":97,"threads_started":1}
I20260620 21:53:11.809060 16096 raft_consensus.cc:482] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:53:11.809314 16096 raft_consensus.cc:504] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:11.810497 16096 leader_election.cc:288] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413), fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194:40759)
Jun 20 21:53:11 dist-test-slave-kk5c krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.194@KRBTEST.COM
Jun 20 21:53:11 dist-test-slave-kk5c krb5kdc[14512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.195@KRBTEST.COM
I20260620 21:53:11.827557 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:11.810686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:37413 (local address 127.0.193.193:36209)
0620 21:53:11.810808 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:11.810832 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:11.810955 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:11.813042 (+ 2087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:11.813048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:11.813590 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:11.814096 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:11.814112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:11.815515 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:11.815520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:11.816670 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:11.816682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:11.817536 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:11.817540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:11.817731 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:11.818559 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:11.818572 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:11.820817 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:11.823618 (+ 2801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:11.823626 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:11.823643 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:11.824256 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:11.824804 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:11.824807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:11.824808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:11.825038 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:11.826041 (+ 1003us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:11.826049 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:11.826434 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 21:53:11.826694 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 21:53:11.827351 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":419,"spinlock_wait_cycles":23296}
I20260620 21:53:11.827710 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:11.810988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:40759 (local address 127.0.193.193:60559)
0620 21:53:11.811431 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:11.811446 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:11.811549 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:11.813042 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:11.813048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:11.813586 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:11.814095 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:11.814112 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:11.815696 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:11.815699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:11.816673 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:11.816683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:11.817648 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:11.817652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:11.817749 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:11.818516 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:11.818533 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:11.820456 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:11.823618 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:11.823658 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:11.823660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:11.824220 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:11.824793 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:11.824797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:11.824799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:11.825038 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:11.826041 (+ 1003us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:11.826049 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:11.826436 (+ 387us) client_negotiation.cc:769] Sending connection context
0620 21:53:11.826715 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 21:53:11.827505 (+ 790us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":145,"spinlock_wait_cycles":86272,"thread_start_us":85,"threads_started":1}
I20260620 21:53:11.827991 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:11.810998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:36209 (local address 127.0.193.195:37413)
0620 21:53:11.811279 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:11.811310 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:11.811368 (+ 58us) server_negotiation.cc:407] Connection header received
0620 21:53:11.812244 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:11.812251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:11.812642 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:11.812959 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:11.814282 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:11.815361 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:11.816889 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:11.817394 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:11.817660 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:11.821033 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:11.821040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:11.821056 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:11.821096 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:11.823269 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:11.824422 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:11.824430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:11.824437 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:11.824648 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:11.825176 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:11.825180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:11.825182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:11.825851 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:11.826000 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:11.826900 (+ 900us) server_negotiation.cc:299] Negotiation successful
0620 21:53:11.827640 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
I20260620 21:53:11.828805 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:11.811155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:60559 (local address 127.0.193.194:40759)
0620 21:53:11.811588 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:11.811607 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:11.811638 (+ 31us) server_negotiation.cc:407] Connection header received
0620 21:53:11.812292 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:11.812297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:11.812650 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:11.812958 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:11.814287 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:11.815515 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:11.816838 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:11.817395 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:11.817601 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:11.820756 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:11.820763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:11.820778 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:11.820813 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:11.823273 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:11.824422 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:11.824430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:11.824437 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:11.824642 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:11.825176 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:11.825180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:11.825182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:11.825851 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:11.826000 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:11.827645 (+ 1645us) server_negotiation.cc:299] Negotiation successful
0620 21:53:11.828629 (+ 984us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":125,"threads_started":1}
W20260620 21:53:11.833261 15712 leader_election.cc:341] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413): Not found: Tablet not found: 26921abdd88748689b2bc89402d55cb0
W20260620 21:53:11.833583 15712 leader_election.cc:341] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194:40759): Not found: Tablet not found: 26921abdd88748689b2bc89402d55cb0
I20260620 21:53:11.833679 15712 leader_election.cc:302] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e19037361c4a449f88e644722ff65f3b; no voters: eabe2a80359f42fab4c94dba012ad0bd, fa52ccbd3c1045ed9caaaf7b025c686d
I20260620 21:53:11.834017 16096 raft_consensus.cc:2726] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:53:12.427827 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:12.427360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:59275)
0620 21:53:12.427660 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:12.427728 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":84,"threads_started":1}
I20260620 21:53:12.573199 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:12.572732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:59313)
0620 21:53:12.573020 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:12.573090 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":75,"threads_started":1}
I20260620 21:53:12.774895 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:12.774451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:59649)
0620 21:53:12.774729 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:12.774795 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
I20260620 21:53:13.429266 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:13.428736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:44655)
0620 21:53:13.429081 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:13.429163 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":76,"threads_started":1}
I20260620 21:53:13.443065 16104 raft_consensus.cc:482] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:53:13.443306 16104 raft_consensus.cc:504] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:13.443704 16104 leader_election.cc:288] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413), fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194:40759)
W20260620 21:53:13.444633 15712 leader_election.cc:341] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413): Not found: Tablet not found: 26921abdd88748689b2bc89402d55cb0
W20260620 21:53:13.444766 15712 leader_election.cc:341] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194:40759): Not found: Tablet not found: 26921abdd88748689b2bc89402d55cb0
I20260620 21:53:13.444824 15712 leader_election.cc:302] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e19037361c4a449f88e644722ff65f3b; no voters: eabe2a80359f42fab4c94dba012ad0bd, fa52ccbd3c1045ed9caaaf7b025c686d
I20260620 21:53:13.445037 16104 raft_consensus.cc:2726] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:53:13.574558 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:13.574004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:48607)
0620 21:53:13.574376 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:13.574442 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":113,"threads_started":1}
I20260620 21:53:13.776717 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:13.776088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:51211)
0620 21:53:13.776448 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:13.776536 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":106,"threads_started":1}
I20260620 21:53:14.430537 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:14.430145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:34091)
0620 21:53:14.430390 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:14.430444 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260620 21:53:14.576159 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:14.575550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:54369)
0620 21:53:14.575920 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:14.575992 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":81,"threads_started":1}
I20260620 21:53:14.778132 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:14.777624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:44533)
0620 21:53:14.777920 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:14.777983 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
I20260620 21:53:15.431897 16110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:15.431432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:46637)
0620 21:53:15.431748 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:15.431804 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":93,"threads_started":1}
I20260620 21:53:15.577456 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:15.577039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:35753)
0620 21:53:15.577294 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:15.577357 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260620 21:53:15.779736 16112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:15.779140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:56811)
0620 21:53:15.779562 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:15.779632 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":178,"threads_started":1}
I20260620 21:53:15.803180 16113 raft_consensus.cc:482] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:53:15.803463 16113 raft_consensus.cc:504] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:15.804088 16113 leader_election.cc:288] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413), fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194:40759)
W20260620 21:53:15.805085 15712 leader_election.cc:341] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413): Not found: Tablet not found: 26921abdd88748689b2bc89402d55cb0
W20260620 21:53:15.805269 15712 leader_election.cc:341] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194:40759): Not found: Tablet not found: 26921abdd88748689b2bc89402d55cb0
I20260620 21:53:15.805333 15712 leader_election.cc:302] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e19037361c4a449f88e644722ff65f3b; no voters: eabe2a80359f42fab4c94dba012ad0bd, fa52ccbd3c1045ed9caaaf7b025c686d
I20260620 21:53:15.805521 16113 raft_consensus.cc:2726] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:53:16.433410 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:16.432936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:53861)
0620 21:53:16.433235 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:16.433304 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":126,"threads_started":1}
I20260620 21:53:16.578683 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:16.578248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:60749)
0620 21:53:16.578513 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:16.578581 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
I20260620 21:53:16.781095 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:16.780596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:56171)
0620 21:53:16.780925 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:16.780992 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260620 21:53:17.434746 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:17.434286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:49453)
0620 21:53:17.434586 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:17.434646 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":72,"threads_started":1}
I20260620 21:53:17.580072 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:17.579551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:46639)
0620 21:53:17.579908 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:17.579978 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":97,"threads_started":1}
I20260620 21:53:17.782387 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:17.781924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:54095)
0620 21:53:17.782228 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:17.782285 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":75,"threads_started":1}
I20260620 21:53:18.436046 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:18.435585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:35985)
0620 21:53:18.435870 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:18.435939 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
I20260620 21:53:18.581429 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:18.580991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:40009)
0620 21:53:18.581259 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:18.581322 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260620 21:53:18.783854 16122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:18.783389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:34029)
0620 21:53:18.783691 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:18.783757 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":86,"threads_started":1}
I20260620 21:53:19.437573 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:19.437107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:34341)
0620 21:53:19.437405 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:19.437471 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":121,"threads_started":1}
I20260620 21:53:19.582820 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:19.582360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:37919)
0620 21:53:19.582657 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:19.582723 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":82,"threads_started":1}
I20260620 21:53:19.785105 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:19.784685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:34221)
0620 21:53:19.784958 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:19.785010 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":106,"threads_started":1}
I20260620 21:53:20.366039 16126 raft_consensus.cc:482] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:53:20.366215 16126 raft_consensus.cc:504] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eabe2a80359f42fab4c94dba012ad0bd" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37413 } } peers { permanent_uuid: "fa52ccbd3c1045ed9caaaf7b025c686d" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 40759 } } peers { permanent_uuid: "e19037361c4a449f88e644722ff65f3b" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 33441 } }
I20260620 21:53:20.366639 16126 leader_election.cc:288] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413), fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194:40759)
W20260620 21:53:20.367617 15712 leader_election.cc:341] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eabe2a80359f42fab4c94dba012ad0bd (127.0.193.195:37413): Not found: Tablet not found: 26921abdd88748689b2bc89402d55cb0
W20260620 21:53:20.367775 15712 leader_election.cc:341] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa52ccbd3c1045ed9caaaf7b025c686d (127.0.193.194:40759): Not found: Tablet not found: 26921abdd88748689b2bc89402d55cb0
I20260620 21:53:20.367830 15712 leader_election.cc:302] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e19037361c4a449f88e644722ff65f3b; no voters: eabe2a80359f42fab4c94dba012ad0bd, fa52ccbd3c1045ed9caaaf7b025c686d
I20260620 21:53:20.368009 16126 raft_consensus.cc:2726] T 26921abdd88748689b2bc89402d55cb0 P e19037361c4a449f88e644722ff65f3b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:53:20.438944 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:20.438529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:40549)
0620 21:53:20.438788 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:20.438841 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260620 21:53:20.584044 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:20.583647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:59201)
0620 21:53:20.583888 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:20.583948 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20260620 21:53:20.786433 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:20.785953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:50301)
0620 21:53:20.786274 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:20.786339 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":115,"threads_started":1}
I20260620 21:53:21.440449 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:21.439955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:58623)
0620 21:53:21.440267 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:21.440335 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":106,"threads_started":1}
I20260620 21:53:21.585382 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:21.584894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:47339)
0620 21:53:21.585199 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:21.585276 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":101,"threads_started":1}
I20260620 21:53:21.788182 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:21.787691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:45939)
0620 21:53:21.787975 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:21.788041 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
I20260620 21:53:22.441689 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:22.441233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:38965)
0620 21:53:22.441526 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:22.441590 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260620 21:53:22.586593 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:22.586153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:50791)
0620 21:53:22.586427 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:22.586498 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
I20260620 21:53:22.789490 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:22.789022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:33397)
0620 21:53:22.789298 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:22.789360 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260620 21:53:23.443159 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:23.442678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:50675)
0620 21:53:23.442982 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:23.443047 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":81,"threads_started":1}
I20260620 21:53:23.588050 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:23.587569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:44459)
0620 21:53:23.587889 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:23.587956 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260620 21:53:23.790892 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:23.790426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:58399)
0620 21:53:23.790723 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:23.790796 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":84,"threads_started":1}
I20260620 21:53:24.444878 16139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:24.444280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:54273)
0620 21:53:24.444628 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:24.444709 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":111,"threads_started":1}
I20260620 21:53:24.589529 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:24.589010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:34211)
0620 21:53:24.589355 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:24.589423 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":150,"threads_started":1}
I20260620 21:53:24.792246 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:24.791797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.195:42277)
0620 21:53:24.792086 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:24.792148 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":90,"threads_started":1}
I20260620 21:53:25.446537 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:25.446117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.193:40501)
0620 21:53:25.446383 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:25.446442 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
I20260620 21:53:25.590948 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:25.590505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:35605 (local address 127.0.193.194:54415)
0620 21:53:25.590771 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:25.590843 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
I20260620 21:53:25.776937 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 15659
I20260620 21:53:25.783401 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 15826
I20260620 21:53:25.790684 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 15959
2026-06-20T21:53:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19736 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:53:25 dist-test-slave-kk5c krb5kdc[16146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:53:25 dist-test-slave-kk5c krb5kdc[16146](info): set up 2 sockets
Jun 20 21:53:25 dist-test-slave-kk5c krb5kdc[16146](info): commencing operation
krb5kdc: starting...
W20260620 21:53:26.859567 775 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:53:26 dist-test-slave-kk5c krb5kdc[16146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:53:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:53:26.999975 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36223
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:36223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:53:27.116381 16159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:27.116674 16159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:27.116717 16159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:27.120939 16159 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 21:53:27.121009 16159 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 21:53:27.121026 16159 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 21:53:27.121040 16159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:53:27.121060 16159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:27.121079 16159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:27.121095 16159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:53:27.121110 16159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:53:27.126855 16159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:36223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:27.128027 16159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:27.129045 16159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:53:27.138828 16159 server_base.cc:1034] running on GCE node
W20260620 21:53:27.138754 16167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:27.138772 16164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:27.138754 16165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:27.139704 16159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:27.140892 16159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:27.142087 16159 hybrid_clock.cc:648] HybridClock initialized: now 1781992407142053 us; error 52 us; skew 500 ppm
Jun 20 21:53:27 dist-test-slave-kk5c krb5kdc[16146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:27.145025 16159 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:53:27.145987 16159 webserver.cc:460] Webserver started at http://127.0.193.254:33287/ using document root <none> and password file <none>
I20260620 21:53:27.146565 16159 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:27.146656 16159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:27.146898 16159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:53:27.149039 16159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "033df915270e41eb8eb89bf91e36ccdf"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "6094e28a6174d5b75e91737caca41bd1"
server_key_iv: "3c07d838668a8c11944abe0f5581700c"
server_key_version: "encryptionkey@0"
I20260620 21:53:27.149605 16159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "033df915270e41eb8eb89bf91e36ccdf"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "6094e28a6174d5b75e91737caca41bd1"
server_key_iv: "3c07d838668a8c11944abe0f5581700c"
server_key_version: "encryptionkey@0"
I20260620 21:53:27.153403 16159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:53:27.155874 16174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:27.157155 16159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:53:27.157315 16159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "033df915270e41eb8eb89bf91e36ccdf"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "6094e28a6174d5b75e91737caca41bd1"
server_key_iv: "3c07d838668a8c11944abe0f5581700c"
server_key_version: "encryptionkey@0"
I20260620 21:53:27.157439 16159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:27.167651 16159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:27.170763 16159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:27.170979 16159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:27.178376 16159 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:36223
I20260620 21:53:27.178826 16226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:36223 every 8 connection(s)
I20260620 21:53:27.179474 16159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:53:27.182339 16227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:27.187422 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 16159
I20260620 21:53:27.187577 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:53:27.187883 775 external_mini_cluster.cc:1428] Setting key 4abec8a04b5eff9d74bb5956868e31fb
I20260620 21:53:27.188171 16227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Bootstrap starting.
I20260620 21:53:27.190629 16227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:27.191568 16227 log.cc:824] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:27.194070 16227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: No bootstrap required, opened a new log
Jun 20 21:53:27 dist-test-slave-kk5c krb5kdc[16146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:27.198804 16227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } }
I20260620 21:53:27.199069 16227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:27.199162 16227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 033df915270e41eb8eb89bf91e36ccdf, State: Initialized, Role: FOLLOWER
I20260620 21:53:27.199700 16227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } }
I20260620 21:53:27.199893 16227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:53:27.200008 16227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:53:27.200147 16227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:27.201148 16227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } }
I20260620 21:53:27.201511 16227 leader_election.cc:302] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 033df915270e41eb8eb89bf91e36ccdf; no voters:
I20260620 21:53:27.201879 16227 leader_election.cc:288] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:53:27.201963 16232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:53:27.202149 16232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 1 LEADER]: Becoming Leader. State: Replica: 033df915270e41eb8eb89bf91e36ccdf, State: Running, Role: LEADER
I20260620 21:53:27.202520 16232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } }
I20260620 21:53:27.203311 16227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:53:27.204437 16234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 033df915270e41eb8eb89bf91e36ccdf. Latest consensus state: current_term: 1 leader_uuid: "033df915270e41eb8eb89bf91e36ccdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } } }
I20260620 21:53:27.204560 16234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: This master's current role is: LEADER
I20260620 21:53:27.204746 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:27.189349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54554 (local address 127.0.193.254:36223)
0620 21:53:27.189779 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:27.189795 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:27.189833 (+ 38us) server_negotiation.cc:407] Connection header received
0620 21:53:27.190471 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:27.190488 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:27.190882 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:27.191169 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:27.192130 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:27.193431 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:27.194578 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:27.195139 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:27.195474 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:27.198744 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:27.198750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:27.198769 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:27.198807 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:27.200959 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:27.201538 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:27.201550 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:27.201555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:27.201723 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:27.202101 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:27.202105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:27.202107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:27.202729 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:27.202973 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:27.203331 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 21:53:27.203668 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":139,"threads_started":1}
I20260620 21:53:27.205786 16233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "033df915270e41eb8eb89bf91e36ccdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } } }
I20260620 21:53:27.205857 16241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:53:27.206146 16233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: This master's current role is: LEADER
I20260620 21:53:27.209796 16241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:53:27.215723 16241 catalog_manager.cc:1347] Generated new cluster ID: 221ac7f7948a4808935438b1d3425324
I20260620 21:53:27.215813 16241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:53:27.226099 16241 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:53:27.227558 16241 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:53:27.250939 16241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Generated new TSK 0
I20260620 21:53:27.251823 16241 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:53:27.284452 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36223
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:53:27.401942 16253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:27.402228 16253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:27.402276 16253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:27.406828 16253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:27.406903 16253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:27.406993 16253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:53:27.412720 16253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:27.413882 16253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:27.414933 16253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:27.425029 16258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:27.425032 16259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:27.425029 16261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:27.425082 16253 server_base.cc:1034] running on GCE node
I20260620 21:53:27.425791 16253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:27.426436 16253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:27.427596 16253 hybrid_clock.cc:648] HybridClock initialized: now 1781992407427560 us; error 53 us; skew 500 ppm
Jun 20 21:53:27 dist-test-slave-kk5c krb5kdc[16146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:27.434296 16253 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:53:27.435407 16253 webserver.cc:460] Webserver started at http://127.0.193.193:38967/ using document root <none> and password file <none>
I20260620 21:53:27.436000 16253 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:27.436067 16253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:27.436254 16253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:53:27.438510 16253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "cc817a1287f71bce6b3104fb4fac0e46"
server_key_iv: "313e310b64f947cf0864fbbd6b4f826f"
server_key_version: "encryptionkey@0"
I20260620 21:53:27.439086 16253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "cc817a1287f71bce6b3104fb4fac0e46"
server_key_iv: "313e310b64f947cf0864fbbd6b4f826f"
server_key_version: "encryptionkey@0"
I20260620 21:53:27.443390 16253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:53:27.446079 16268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:27.447557 16253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:27.447687 16253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "cc817a1287f71bce6b3104fb4fac0e46"
server_key_iv: "313e310b64f947cf0864fbbd6b4f826f"
server_key_version: "encryptionkey@0"
I20260620 21:53:27.447788 16253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:27.464265 16253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:27.467744 16253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:27.467911 16253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:27.468464 16253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:27.469277 16253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:53:27.469333 16253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:27.469376 16253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:53:27.469393 16253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:27.480417 16253 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:35883
I20260620 21:53:27.480440 16381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:35883 every 8 connection(s)
I20260620 21:53:27.481624 16253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
Jun 20 21:53:27 dist-test-slave-kk5c krb5kdc[16146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:27.491856 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 16253
I20260620 21:53:27.492005 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:53:27.492420 775 external_mini_cluster.cc:1428] Setting key e6ab5038addd31e4411b2ed16586246c
I20260620 21:53:27.496995 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:27.483730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:48521 (local address 127.0.193.254:36223)
0620 21:53:27.483924 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:27.483929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:27.484960 (+ 1031us) server_negotiation.cc:407] Connection header received
0620 21:53:27.485853 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:27.485856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:27.485906 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:27.485987 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:27.487296 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:27.487891 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:27.489078 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:27.489336 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:27.489427 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:27.492395 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:27.492399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:27.492401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:27.492430 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:27.494478 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:27.495103 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:27.495107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:27.495109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:27.495249 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:27.495782 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:27.495786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:27.495788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:27.495992 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:27.496106 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:27.496685 (+ 579us) server_negotiation.cc:299] Negotiation successful
0620 21:53:27.496825 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 21:53:27.497936 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:27.484049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:48521)
0620 21:53:27.484801 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:27.484846 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:27.485611 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:27.486154 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:27.486160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:27.486547 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:27.487087 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:27.487100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:27.488012 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:27.488015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:27.488952 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:27.488961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:27.489476 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:27.489479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:27.489617 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:27.490286 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:27.490303 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:27.492183 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:27.494610 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:27.494617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:27.494637 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:27.494994 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:27.495444 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:27.495448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:27.495450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:27.495669 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:27.496110 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:27.496120 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:27.496435 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 21:53:27.496645 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 21:53:27.497046 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":129,"threads_started":1}
I20260620 21:53:27.499471 16382 heartbeater.cc:347] Connected to a master server at 127.0.193.254:36223
I20260620 21:53:27.499749 16382 heartbeater.cc:464] Registering TS with master...
I20260620 21:53:27.500288 16382 heartbeater.cc:511] Master 127.0.193.254:36223 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
I20260620 21:53:27.502058 16191 ts_manager.cc:194] Registered new tserver with Master: 53cd1d8a03dd401b913ec3b6d0fad6c8 (127.0.193.193:35883)
I20260620 21:53:27.503466 16191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:48521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:53:27.527079 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36223
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:53:27.660983 16387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:27.661306 16387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:27.661358 16387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:27.666311 16387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:27.666400 16387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:27.666463 16387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:53:27.672101 16387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:27.673233 16387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:27.674301 16387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:27.687530 16393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:27.687522 16392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:27.688549 16395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:27.688846 16387 server_base.cc:1034] running on GCE node
I20260620 21:53:27.690026 16387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:27.691152 16387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:27.692430 16387 hybrid_clock.cc:648] HybridClock initialized: now 1781992407692403 us; error 33 us; skew 500 ppm
Jun 20 21:53:27 dist-test-slave-kk5c krb5kdc[16146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:27.696907 16387 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:53:27.698058 16387 webserver.cc:460] Webserver started at http://127.0.193.194:42321/ using document root <none> and password file <none>
I20260620 21:53:27.698879 16387 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:27.698977 16387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:27.699262 16387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:53:27.702368 16387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "dabb641223ee4852bfb2b7e67242997a"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "9d54f5a37d68984040059dbbf9a94190"
server_key_iv: "1b5e00122e3774fd5ac6fdd3278359dc"
server_key_version: "encryptionkey@0"
I20260620 21:53:27.703493 16387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "dabb641223ee4852bfb2b7e67242997a"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "9d54f5a37d68984040059dbbf9a94190"
server_key_iv: "1b5e00122e3774fd5ac6fdd3278359dc"
server_key_version: "encryptionkey@0"
I20260620 21:53:27.709890 16387 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260620 21:53:27.713755 16402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:27.715864 16387 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:53:27.716030 16387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "dabb641223ee4852bfb2b7e67242997a"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "9d54f5a37d68984040059dbbf9a94190"
server_key_iv: "1b5e00122e3774fd5ac6fdd3278359dc"
server_key_version: "encryptionkey@0"
I20260620 21:53:27.716216 16387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:27.753192 16387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:27.756788 16387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:27.757002 16387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:27.757608 16387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:27.758441 16387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:53:27.758487 16387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:27.758548 16387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:53:27.758580 16387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:27.767839 16387 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:33481
I20260620 21:53:27.767856 16515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:33481 every 8 connection(s)
I20260620 21:53:27.768796 16387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:53:27.775524 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 16387
I20260620 21:53:27.775671 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:53:27.775957 775 external_mini_cluster.cc:1428] Setting key b77edf895742b26a6a2fb791d3836bba
Jun 20 21:53:27 dist-test-slave-kk5c krb5kdc[16146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:27.784036 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:27.770905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:45247 (local address 127.0.193.254:36223)
0620 21:53:27.771059 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:27.771064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:27.772363 (+ 1299us) server_negotiation.cc:407] Connection header received
0620 21:53:27.773264 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:27.773266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:27.773325 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:27.773422 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:27.774783 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:27.775471 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:27.776983 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:27.777222 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:27.777338 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:27.780262 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:27.780266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:27.780269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:27.780305 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:27.781749 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:27.782321 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:27.782325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:27.782327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:27.782408 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:27.782783 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:27.782786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:27.782788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:27.782988 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:27.783099 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:27.783726 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 21:53:27.783868 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:53:27.784894 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:27.771407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:45247)
0620 21:53:27.772155 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:27.772198 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:27.773012 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:27.773645 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:27.773652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:27.774068 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:27.774617 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:27.774631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:27.775686 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:27.775689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:27.776855 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:27.776866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:27.777393 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:27.777397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:27.777578 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:27.778349 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:27.778368 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:27.780064 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:27.781884 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:27.781890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:27.781910 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:27.782207 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:27.782500 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:27.782504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:27.782506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:27.782674 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:27.783099 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:27.783106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:27.783449 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 21:53:27.783679 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 21:53:27.784038 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":149,"threads_started":1}
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
I20260620 21:53:27.786237 16516 heartbeater.cc:347] Connected to a master server at 127.0.193.254:36223
I20260620 21:53:27.786592 16516 heartbeater.cc:464] Registering TS with master...
I20260620 21:53:27.787135 16516 heartbeater.cc:511] Master 127.0.193.254:36223 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
I20260620 21:53:27.788586 16191 ts_manager.cc:194] Registered new tserver with Master: dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481)
I20260620 21:53:27.789271 16191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:45247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:53:27.807529 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36223
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:53:27.923163 16521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:27.923590 16521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:27.923667 16521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:27.928414 16521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:27.928495 16521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:27.928560 16521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:53:27.934435 16521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:27.936014 16521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:27.937191 16521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:27.947575 16526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:27.947783 16527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:27.947728 16521 server_base.cc:1034] running on GCE node
W20260620 21:53:27.947662 16529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:27.948405 16521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:27.949090 16521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:27.950307 16521 hybrid_clock.cc:648] HybridClock initialized: now 1781992407950271 us; error 43 us; skew 500 ppm
Jun 20 21:53:27 dist-test-slave-kk5c krb5kdc[16146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:27.953465 16521 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:53:27.954388 16521 webserver.cc:460] Webserver started at http://127.0.193.195:33775/ using document root <none> and password file <none>
I20260620 21:53:27.954917 16521 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:27.954977 16521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:27.955152 16521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:53:27.957301 16521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "78b7ec3c91454fb08fec7e1824118efc"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "e37955098c20cdf396ae88ea47461d6e"
server_key_iv: "d5a98f232339396bb9445b9c75da21ae"
server_key_version: "encryptionkey@0"
I20260620 21:53:27.957835 16521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "78b7ec3c91454fb08fec7e1824118efc"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "e37955098c20cdf396ae88ea47461d6e"
server_key_iv: "d5a98f232339396bb9445b9c75da21ae"
server_key_version: "encryptionkey@0"
I20260620 21:53:27.962035 16521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 21:53:27.965067 16536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:27.966557 16521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:53:27.966713 16521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "78b7ec3c91454fb08fec7e1824118efc"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "e37955098c20cdf396ae88ea47461d6e"
server_key_iv: "d5a98f232339396bb9445b9c75da21ae"
server_key_version: "encryptionkey@0"
I20260620 21:53:27.966845 16521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:27.987658 16521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:27.991070 16521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:27.991320 16521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:27.991930 16521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:27.992791 16521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:53:27.992839 16521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:27.992902 16521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:53:27.992933 16521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:28.003080 16521 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:46345
I20260620 21:53:28.003139 16649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:46345 every 8 connection(s)
I20260620 21:53:28.004268 16521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:53:28.005110 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 16521
I20260620 21:53:28.005250 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:53:28.005558 775 external_mini_cluster.cc:1428] Setting key c9537f23a60ae7d9bc84a2c06d6c3744
Jun 20 21:53:28 dist-test-slave-kk5c krb5kdc[16146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:28.018630 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.006321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:56471 (local address 127.0.193.254:36223)
0620 21:53:28.006478 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:28.006483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:28.007795 (+ 1312us) server_negotiation.cc:407] Connection header received
0620 21:53:28.008725 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:28.008728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:28.008783 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:28.008865 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:28.010094 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.010641 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.011874 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.012112 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.012230 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:28.014921 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:28.014924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:28.014927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:28.014953 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:28.016447 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:28.016976 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:28.016979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:28.016980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:28.017060 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:28.017420 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:28.017423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:28.017424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:28.017634 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:28.017726 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:28.018322 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 21:53:28.018462 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:53:28.019659 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.006834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:56471)
0620 21:53:28.007637 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:28.007703 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:28.008501 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:28.009040 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:28.009047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:28.009438 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:28.009944 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:28.009959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.010775 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.010779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:28.011750 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:28.011760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.012275 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.012280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:28.012453 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:28.013107 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:28.013125 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:28.014730 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:28.016568 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:28.016573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:28.016588 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:28.016878 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:28.017150 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:28.017152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:28.017154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:28.017313 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:28.017744 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:28.017749 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:28.018023 (+ 274us) client_negotiation.cc:769] Sending connection context
0620 21:53:28.018276 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 21:53:28.018654 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":178,"threads_started":1}
I20260620 21:53:28.020773 16650 heartbeater.cc:347] Connected to a master server at 127.0.193.254:36223
I20260620 21:53:28.021021 16650 heartbeater.cc:464] Registering TS with master...
I20260620 21:53:28.021525 16650 heartbeater.cc:511] Master 127.0.193.254:36223 requested a full tablet report, sending...
I20260620 21:53:28.022671 16191 ts_manager.cc:194] Registered new tserver with Master: 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345)
I20260620 21:53:28.023336 16191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:56471
I20260620 21:53:28.031747 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:53:28.042093 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.034019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.0.193.254:36223)
0620 21:53:28.034187 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:28.034192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:28.034354 (+ 162us) server_negotiation.cc:407] Connection header received
0620 21:53:28.034505 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:28.034509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:28.034560 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:28.034647 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:28.035500 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.036048 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.036942 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.037180 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.037287 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:28.038839 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:28.038843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:28.038846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:28.038878 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:28.040329 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:28.040790 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:28.040795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:28.040797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:28.040882 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:28.041156 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:28.041159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:28.041161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:28.041404 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:28.041547 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:28.041785 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 21:53:28.041907 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 21:53:28.045439 16191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:53:28.048715 16191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:53:28.064424 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.058963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.0.193.194:33481)
0620 21:53:28.059855 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:28.059870 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:28.059893 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:53:28.059971 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:28.059978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:28.060179 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:28.060334 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:28.060495 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.061653 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.063065 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.063727 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.063852 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:28.064017 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:28.064146 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 21:53:28.064254 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":109,"threads_started":1}
I20260620 21:53:28.064414 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.058810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:33481 (local address 127.0.0.1:39402)
0620 21:53:28.059385 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:28.059398 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:28.059475 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:28.060303 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:28.060305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:28.060323 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:28.060409 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:28.060416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.061787 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.061791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:28.062953 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:28.062959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.063853 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.063857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:28.063962 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:28.063998 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 21:53:28.064117 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 21:53:28.064254 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":28,"thread_start_us":95,"threads_started":1}
I20260620 21:53:28.064816 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.058497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.0.193.193:35883)
0620 21:53:28.058901 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:28.058911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:28.059469 (+ 558us) server_negotiation.cc:407] Connection header received
0620 21:53:28.059693 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:28.059698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:28.059880 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:28.060018 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:28.060252 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.061333 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.062948 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.063880 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.064001 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:28.064168 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:28.064513 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 21:53:28.064646 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":100,"threads_started":1}
I20260620 21:53:28.065167 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.057972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:46345 (local address 127.0.0.1:35040)
0620 21:53:28.058448 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:28.058510 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:28.058634 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:28.059556 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:28.059562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:28.059581 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:28.059751 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:28.059757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.061263 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.061267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:28.063007 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:28.063018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.064430 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.064436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:28.064719 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:28.064744 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:53:28.064837 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 21:53:28.065014 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":213,"thread_start_us":109,"threads_started":1}
I20260620 21:53:28.065191 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.058090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35040 (local address 127.0.193.195:46345)
0620 21:53:28.058817 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:28.058829 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:28.058853 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:53:28.058930 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:28.058936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:28.059163 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:28.059361 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:28.059881 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.061114 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.063156 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.064114 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.064229 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:28.064366 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:28.064908 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 21:53:28.065028 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":134,"threads_started":1}
I20260620 21:53:28.065593 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.058150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:35883 (local address 127.0.0.1:43240)
0620 21:53:28.059369 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:28.059384 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:28.059547 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:28.059995 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:28.059998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:28.060017 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:28.060119 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:28.060126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.061438 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.061443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:28.062814 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:28.062825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.064240 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.064244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:28.064341 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:28.064365 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:53:28.065037 (+ 672us) client_negotiation.cc:240] Negotiation successful
0620 21:53:28.065117 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"mutex_wait_us":62,"thread_start_us":116,"threads_started":1}
I20260620 21:53:28.068979 16316 tablet_service.cc:1467] Processing CreateTablet for tablet e4195f8eef9f44aab87dc4e41812f726 (DEFAULT_TABLE table=test-table [id=62ba140b47e94f85bb04206fabe27ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:53:28.068977 16450 tablet_service.cc:1467] Processing CreateTablet for tablet e4195f8eef9f44aab87dc4e41812f726 (DEFAULT_TABLE table=test-table [id=62ba140b47e94f85bb04206fabe27ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:53:28.070461 16316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4195f8eef9f44aab87dc4e41812f726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:28.070461 16450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4195f8eef9f44aab87dc4e41812f726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:28.070137 16584 tablet_service.cc:1467] Processing CreateTablet for tablet e4195f8eef9f44aab87dc4e41812f726 (DEFAULT_TABLE table=test-table [id=62ba140b47e94f85bb04206fabe27ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:53:28.072036 16584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4195f8eef9f44aab87dc4e41812f726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:28.078277 16667 tablet_bootstrap.cc:492] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc: Bootstrap starting.
I20260620 21:53:28.079231 16669 tablet_bootstrap.cc:492] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a: Bootstrap starting.
I20260620 21:53:28.080986 16667 tablet_bootstrap.cc:654] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:28.081352 16669 tablet_bootstrap.cc:654] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:28.082193 16667 log.cc:824] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:28.082193 16669 log.cc:824] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:28.082931 16668 tablet_bootstrap.cc:492] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: Bootstrap starting.
I20260620 21:53:28.084151 16667 tablet_bootstrap.cc:492] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc: No bootstrap required, opened a new log
I20260620 21:53:28.084362 16667 ts_tablet_manager.cc:1397] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:53:28.084878 16668 tablet_bootstrap.cc:654] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:28.085547 16668 log.cc:824] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:28.087117 16669 tablet_bootstrap.cc:492] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a: No bootstrap required, opened a new log
I20260620 21:53:28.087358 16669 ts_tablet_manager.cc:1397] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 21:53:28.087447 16668 tablet_bootstrap.cc:492] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: No bootstrap required, opened a new log
I20260620 21:53:28.087610 16668 ts_tablet_manager.cc:1397] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:53:28.087697 16667 raft_consensus.cc:348] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:28.087960 16667 raft_consensus.cc:374] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:28.088013 16667 raft_consensus.cc:729] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b7ec3c91454fb08fec7e1824118efc, State: Initialized, Role: FOLLOWER
I20260620 21:53:28.088649 16667 consensus_queue.cc:260] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:28.089681 16667 ts_tablet_manager.cc:1428] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 21:53:28.090081 16650 heartbeater.cc:503] Master 127.0.193.254:36223 was elected leader, sending a full tablet report...
I20260620 21:53:28.091578 16669 raft_consensus.cc:348] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:28.091885 16669 raft_consensus.cc:374] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:28.091948 16669 raft_consensus.cc:729] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabb641223ee4852bfb2b7e67242997a, State: Initialized, Role: FOLLOWER
I20260620 21:53:28.092417 16669 consensus_queue.cc:260] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:28.092839 16668 raft_consensus.cc:348] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:28.093081 16668 raft_consensus.cc:374] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:28.093143 16668 raft_consensus.cc:729] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53cd1d8a03dd401b913ec3b6d0fad6c8, State: Initialized, Role: FOLLOWER
I20260620 21:53:28.093174 16516 heartbeater.cc:503] Master 127.0.193.254:36223 was elected leader, sending a full tablet report...
I20260620 21:53:28.093479 16669 ts_tablet_manager.cc:1428] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:53:28.093549 16668 consensus_queue.cc:260] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:28.094166 16382 heartbeater.cc:503] Master 127.0.193.254:36223 was elected leader, sending a full tablet report...
I20260620 21:53:28.094496 16668 ts_tablet_manager.cc:1428] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 21:53:28.141923 16675 raft_consensus.cc:482] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:53:28.142145 16675 raft_consensus.cc:504] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:28.143378 16675 leader_election.cc:288] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345), dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481)
I20260620 21:53:28.149391 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.144232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:57997 (local address 127.0.193.194:33481)
0620 21:53:28.144367 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:28.144372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:28.144536 (+ 164us) server_negotiation.cc:407] Connection header received
0620 21:53:28.144622 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:28.144624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:28.144672 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:28.144731 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:28.145005 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.145723 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.147699 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.148519 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.148673 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:28.148733 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:28.149157 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 21:53:28.149270 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:53:28.149391 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.143712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:42119 (local address 127.0.193.195:46345)
0620 21:53:28.143868 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:28.143874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:28.144061 (+ 187us) server_negotiation.cc:407] Connection header received
0620 21:53:28.144294 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:28.144297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:28.144359 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:28.144426 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:28.144855 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.145644 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.147699 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.148521 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.148673 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:28.148733 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:28.149157 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 21:53:28.149248 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:53:28.149394 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.143597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:46345 (local address 127.0.193.193:42119)
0620 21:53:28.143961 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:28.143975 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:28.144155 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:28.144461 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:28.144467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:28.144493 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:28.144727 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:28.144739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.145765 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.145789 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:28.147487 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:28.147501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.148705 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.148708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:28.148961 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:28.149005 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 21:53:28.149095 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 21:53:28.149188 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":234,"spinlock_wait_cycles":3840,"thread_start_us":151,"threads_started":1}
I20260620 21:53:28.149394 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.144073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:33481 (local address 127.0.193.193:57997)
0620 21:53:28.144430 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:28.144444 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:28.144544 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:28.144760 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:28.144766 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:28.144788 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:28.144902 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:28.144907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.145821 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.145825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:28.147487 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:28.147501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.148696 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.148700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:28.148957 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:28.149003 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 21:53:28.149138 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 21:53:28.149187 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":234,"spinlock_wait_cycles":1408,"thread_start_us":117,"threads_started":1}
I20260620 21:53:28.150277 16470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4195f8eef9f44aab87dc4e41812f726" candidate_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabb641223ee4852bfb2b7e67242997a" is_pre_election: true
I20260620 21:53:28.150301 16604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4195f8eef9f44aab87dc4e41812f726" candidate_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b7ec3c91454fb08fec7e1824118efc" is_pre_election: true
I20260620 21:53:28.150616 16604 raft_consensus.cc:2454] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53cd1d8a03dd401b913ec3b6d0fad6c8 in term 0.
I20260620 21:53:28.150615 16470 raft_consensus.cc:2454] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53cd1d8a03dd401b913ec3b6d0fad6c8 in term 0.
I20260620 21:53:28.151139 16271 leader_election.cc:302] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53cd1d8a03dd401b913ec3b6d0fad6c8, 78b7ec3c91454fb08fec7e1824118efc; no voters:
I20260620 21:53:28.151429 16675 raft_consensus.cc:2781] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:53:28.151547 16675 raft_consensus.cc:482] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:53:28.151623 16675 raft_consensus.cc:3037] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:28.152957 16675 raft_consensus.cc:504] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:28.153450 16675 leader_election.cc:288] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 1 election: Requested vote from peers 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345), dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481)
I20260620 21:53:28.153954 16604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4195f8eef9f44aab87dc4e41812f726" candidate_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b7ec3c91454fb08fec7e1824118efc"
I20260620 21:53:28.153985 16470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4195f8eef9f44aab87dc4e41812f726" candidate_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabb641223ee4852bfb2b7e67242997a"
I20260620 21:53:28.154105 16604 raft_consensus.cc:3037] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:28.154138 16470 raft_consensus.cc:3037] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:28.155315 16470 raft_consensus.cc:2454] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53cd1d8a03dd401b913ec3b6d0fad6c8 in term 1.
I20260620 21:53:28.155328 16604 raft_consensus.cc:2454] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53cd1d8a03dd401b913ec3b6d0fad6c8 in term 1.
I20260620 21:53:28.155704 16271 leader_election.cc:302] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53cd1d8a03dd401b913ec3b6d0fad6c8, 78b7ec3c91454fb08fec7e1824118efc; no voters:
I20260620 21:53:28.155907 16675 raft_consensus.cc:2781] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:53:28.156186 16675 raft_consensus.cc:686] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 LEADER]: Becoming Leader. State: Replica: 53cd1d8a03dd401b913ec3b6d0fad6c8, State: Running, Role: LEADER
I20260620 21:53:28.156569 16675 consensus_queue.cc:237] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:28.160099 16190 catalog_manager.cc:5495] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53cd1d8a03dd401b913ec3b6d0fad6c8 (127.0.193.193). New cstate: current_term: 1 leader_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } health_report { overall_health: HEALTHY } } }
I20260620 21:53:28.191747 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.186905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.0.193.193:35883)
0620 21:53:28.187055 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:28.187061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:28.187080 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:53:28.187177 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:28.187180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:28.187286 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:28.187403 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:53:28.187683 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.188609 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.189561 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:28.189899 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:28.190002 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:28.190591 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:53:28.191169 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:53:28.191330 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:28.191530 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 21:53:28.191599 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:53:28.199741 16604 raft_consensus.cc:1261] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc [term 1 FOLLOWER]: Refusing update from remote peer 53cd1d8a03dd401b913ec3b6d0fad6c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:53:28.199846 16470 raft_consensus.cc:1261] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a [term 1 FOLLOWER]: Refusing update from remote peer 53cd1d8a03dd401b913ec3b6d0fad6c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:53:28.200771 16678 consensus_queue.cc:1035] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:53:28.201107 16675 consensus_queue.cc:1035] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:53:28.209664 16683 mvcc.cc:205] Tried to move back new op lower bound from 7299040903976312832 to 7299040903812513792. Current Snapshot: MvccSnapshot[applied={T|T < 7299040903976312832}]
I20260620 21:53:28.210448 16682 mvcc.cc:205] Tried to move back new op lower bound from 7299040903976312832 to 7299040903812513792. Current Snapshot: MvccSnapshot[applied={T|T < 7299040903976312832}]
I20260620 21:53:28.219798 16189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:53:28.220055 16189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:53:28.222831 16189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Sending DeleteTablet for 3 replicas of tablet e4195f8eef9f44aab87dc4e41812f726
I20260620 21:53:28.223909 16584 tablet_service.cc:1514] Processing DeleteTablet for tablet e4195f8eef9f44aab87dc4e41812f726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:28 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:35040
I20260620 21:53:28.223903 16316 tablet_service.cc:1514] Processing DeleteTablet for tablet e4195f8eef9f44aab87dc4e41812f726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:28 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:43240
I20260620 21:53:28.223903 16450 tablet_service.cc:1514] Processing DeleteTablet for tablet e4195f8eef9f44aab87dc4e41812f726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:28 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:39402
I20260620 21:53:28.224515 16690 tablet_replica.cc:330] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a: stopping tablet replica
I20260620 21:53:28.224736 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 16253
I20260620 21:53:28.224893 16690 raft_consensus.cc:2229] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:53:28.225253 16690 raft_consensus.cc:2258] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:53:28.225449 16692 tablet_replica.cc:330] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc: stopping tablet replica
I20260620 21:53:28.225903 16692 raft_consensus.cc:2229] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:53:28.226259 16692 raft_consensus.cc:2258] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:53:28.227252 16690 ts_tablet_manager.cc:1905] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:53:28.227881 16692 ts_tablet_manager.cc:1905] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:53:28.230432 16692 ts_tablet_manager.cc:1918] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:53:28.230583 16692 log.cc:1196] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/e4195f8eef9f44aab87dc4e41812f726
I20260620 21:53:28.231036 16692 ts_tablet_manager.cc:1939] T e4195f8eef9f44aab87dc4e41812f726 P 78b7ec3c91454fb08fec7e1824118efc: Deleting consensus metadata
I20260620 21:53:28.231011 16690 ts_tablet_manager.cc:1918] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:53:28.231268 16690 log.cc:1196] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/e4195f8eef9f44aab87dc4e41812f726
I20260620 21:53:28.231607 16690 ts_tablet_manager.cc:1939] T e4195f8eef9f44aab87dc4e41812f726 P dabb641223ee4852bfb2b7e67242997a: Deleting consensus metadata
I20260620 21:53:28.232860 16177 catalog_manager.cc:4841] TS 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345): tablet e4195f8eef9f44aab87dc4e41812f726 (table test-table [id=62ba140b47e94f85bb04206fabe27ee1]) successfully deleted
I20260620 21:53:28.233147 16177 catalog_manager.cc:4841] TS dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481): tablet e4195f8eef9f44aab87dc4e41812f726 (table test-table [id=62ba140b47e94f85bb04206fabe27ee1]) successfully deleted
W20260620 21:53:28.233834 16176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:35883 (error 108)
I20260620 21:53:28.233947 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 16387
I20260620 21:53:28.235374 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:28.235022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:35883 (local address 127.0.0.1:43264)
0620 21:53:28.235162 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:28.235262 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 21:53:28.235556 16176 catalog_manager.cc:4584] TS 53cd1d8a03dd401b913ec3b6d0fad6c8 (127.0.193.193:35883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4195f8eef9f44aab87dc4e41812f726: Network error: Client connection negotiation failed: client connection to 127.0.193.193:35883: connect: Connection refused (error 111)
I20260620 21:53:28.240658 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 16521
I20260620 21:53:28.246969 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 16159
I20260620 21:53:48.254675 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36223
--webserver_interface=127.0.193.254
--webserver_port=33287
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:36223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:53:48.370487 16694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:48.370872 16694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:48.370961 16694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:48.375298 16694 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 21:53:48.375418 16694 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 21:53:48.375448 16694 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 21:53:48.375473 16694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:53:48.375516 16694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:48.375553 16694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:48.375576 16694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:53:48.375610 16694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:53:48.382745 16694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:36223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:48.384029 16694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:48.385139 16694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:53:48.399840 16694 server_base.cc:1034] running on GCE node
W20260620 21:53:48.399744 16701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:48.399739 16700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:48.399751 16703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:48.400744 16694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:48.402081 16694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:48.403293 16694 hybrid_clock.cc:648] HybridClock initialized: now 1781992428403275 us; error 34 us; skew 500 ppm
Jun 20 21:53:48 dist-test-slave-kk5c krb5kdc[16146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:48.406396 16694 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:53:48.407501 16694 webserver.cc:460] Webserver started at http://127.0.193.254:33287/ using document root <none> and password file <none>
I20260620 21:53:48.408121 16694 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:48.408241 16694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:48.412559 16694 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:53:48.414937 16710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:48.416574 16694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:48.416772 16694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "033df915270e41eb8eb89bf91e36ccdf"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "6094e28a6174d5b75e91737caca41bd1"
server_key_iv: "3c07d838668a8c11944abe0f5581700c"
server_key_version: "encryptionkey@0"
I20260620 21:53:48.417460 16694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:48.442997 16694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:48.446502 16694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:48.446738 16694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:48.454236 16694 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:36223
I20260620 21:53:48.454536 16762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:36223 every 8 connection(s)
I20260620 21:53:48.455467 16694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:53:48.460026 16763 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:53:48.461858 16763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Bootstrap starting.
I20260620 21:53:48.462786 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 16694
I20260620 21:53:48.463403 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:35883
--local_ip_for_outbound_sockets=127.0.193.193
--tserver_master_addrs=127.0.193.254:36223
--webserver_port=38967
--webserver_interface=127.0.193.193
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:53:48.480008 16763 log.cc:824] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:48.489050 16763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:53:48.489475 16763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Bootstrap complete.
I20260620 21:53:48.493589 16763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } }
I20260620 21:53:48.494040 16763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 033df915270e41eb8eb89bf91e36ccdf, State: Initialized, Role: FOLLOWER
I20260620 21:53:48.494540 16763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } }
I20260620 21:53:48.494695 16763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:53:48.494791 16763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:53:48.494926 16763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 1 FOLLOWER]: Advancing to term 2
I20260620 21:53:48.496963 16763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } }
I20260620 21:53:48.497313 16763 leader_election.cc:302] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 033df915270e41eb8eb89bf91e36ccdf; no voters:
I20260620 21:53:48.497630 16763 leader_election.cc:288] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 21:53:48.497788 16767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 2 FOLLOWER]: Leader election won for term 2
I20260620 21:53:48.498131 16767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 2 LEADER]: Becoming Leader. State: Replica: 033df915270e41eb8eb89bf91e36ccdf, State: Running, Role: LEADER
I20260620 21:53:48.498520 16767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } }
I20260620 21:53:48.498903 16763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:53:48.500293 16769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 033df915270e41eb8eb89bf91e36ccdf. Latest consensus state: current_term: 2 leader_uuid: "033df915270e41eb8eb89bf91e36ccdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } } }
I20260620 21:53:48.500433 16769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: This master's current role is: LEADER
I20260620 21:53:48.500480 16768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "033df915270e41eb8eb89bf91e36ccdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } } }
I20260620 21:53:48.500582 16768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: This master's current role is: LEADER
I20260620 21:53:48.505151 16782 catalog_manager.cc:1259] Loaded cluster ID: 221ac7f7948a4808935438b1d3425324
I20260620 21:53:48.505230 16782 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: loading cluster ID for follower catalog manager: success
I20260620 21:53:48.508365 16782 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: acquiring CA information for follower catalog manager: success
I20260620 21:53:48.510143 16782 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:53:48.510318 16777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:53:48.512898 16777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:53:48.513417 16777 catalog_manager.cc:1259] Loaded cluster ID: 221ac7f7948a4808935438b1d3425324
I20260620 21:53:48.513499 16777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:53:48.514348 16777 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:53:48.514875 16777 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Loaded TSK: 0
I20260620 21:53:48.530220 16777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Generated new TSK 1
I20260620 21:53:48.530550 16777 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:53:48.591452 16765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:48.591814 16765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:48.591909 16765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:48.596657 16765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:48.596758 16765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:48.596855 16765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:53:48.603711 16765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:35883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=38967
--tserver_master_addrs=127.0.193.254:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:48.604862 16765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:48.606088 16765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:48.617655 16793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:48.617643 16790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:48.617648 16791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:48.618793 16765 server_base.cc:1034] running on GCE node
I20260620 21:53:48.619321 16765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:48.620035 16765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:48.621238 16765 hybrid_clock.cc:648] HybridClock initialized: now 1781992428621199 us; error 51 us; skew 500 ppm
Jun 20 21:53:48 dist-test-slave-kk5c krb5kdc[16146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:48.624491 16765 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:53:48.625509 16765 webserver.cc:460] Webserver started at http://127.0.193.193:38967/ using document root <none> and password file <none>
I20260620 21:53:48.626189 16765 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:48.626315 16765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:48.630716 16765 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:53:48.633585 16800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:48.635133 16765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:53:48.635320 16765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "cc817a1287f71bce6b3104fb4fac0e46"
server_key_iv: "313e310b64f947cf0864fbbd6b4f826f"
server_key_version: "encryptionkey@0"
I20260620 21:53:48.635921 16765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:48.657004 16765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:48.660678 16765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:48.660890 16765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:48.661525 16765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:48.662683 16807 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 21:53:48.668262 16765 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 21:53:48.668363 16765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 21:53:48.668458 16765 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 21:53:48.670217 16765 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 21:53:48.670295 16765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 21:53:48.670624 16807 tablet_bootstrap.cc:492] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: Bootstrap starting.
I20260620 21:53:48.681321 16765 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:35883
I20260620 21:53:48.681385 16914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:35883 every 8 connection(s)
I20260620 21:53:48.682363 16765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:53:48.690785 16807 log.cc:824] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:48.691381 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 16765
I20260620 21:53:48.691855 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:33481
--local_ip_for_outbound_sockets=127.0.193.194
--tserver_master_addrs=127.0.193.254:36223
--webserver_port=42321
--webserver_interface=127.0.193.194
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 21:53:48 dist-test-slave-kk5c krb5kdc[16146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:48.701937 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:48.684533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:47495 (local address 127.0.193.254:36223)
0620 21:53:48.685106 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:48.685133 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:48.685484 (+ 351us) server_negotiation.cc:407] Connection header received
0620 21:53:48.686747 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:48.686767 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:48.687293 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:48.687810 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:48.688936 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:48.690394 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:48.692150 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:48.692679 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:48.692872 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:48.695829 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:48.695834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:48.695848 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:48.695879 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:48.697940 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:48.698703 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:48.698710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:48.698716 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:48.698890 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:48.699363 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:48.699366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:48.699368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:48.699901 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:48.700041 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:48.700718 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 21:53:48.701000 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":275,"threads_started":1}
I20260620 21:53:48.701961 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:48.684441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:47495)
0620 21:53:48.685262 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:48.685317 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:48.686208 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:48.687862 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:48.687869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:48.688251 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:48.688739 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:48.688754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:48.690580 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:48.690583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:48.691964 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:48.691981 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:48.692845 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:48.692848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:48.693035 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:48.693729 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:48.693754 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:48.695590 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:48.698118 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:48.698122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:48.698133 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:48.698555 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:48.699036 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:48.699038 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:48.699041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:48.699259 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:48.700051 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:48.700061 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:48.700338 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 21:53:48.700577 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 21:53:48.701037 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":275,"threads_started":1}
I20260620 21:53:48.702332 16807 tablet_bootstrap.cc:492] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:53:48.702872 16807 tablet_bootstrap.cc:492] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: Bootstrap complete.
I20260620 21:53:48.704162 16807 ts_tablet_manager.cc:1397] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.006s
I20260620 21:53:48.705113 16915 heartbeater.cc:347] Connected to a master server at 127.0.193.254:36223
I20260620 21:53:48.705351 16915 heartbeater.cc:464] Registering TS with master...
I20260620 21:53:48.705914 16915 heartbeater.cc:511] Master 127.0.193.254:36223 requested a full tablet report, sending...
I20260620 21:53:48.707229 16807 raft_consensus.cc:348] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:48.707755 16807 raft_consensus.cc:729] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53cd1d8a03dd401b913ec3b6d0fad6c8, State: Initialized, Role: FOLLOWER
I20260620 21:53:48.708299 16807 consensus_queue.cc:260] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:48.709110 16727 ts_manager.cc:194] Registered new tserver with Master: 53cd1d8a03dd401b913ec3b6d0fad6c8 (127.0.193.193:35883)
I20260620 21:53:48.709743 16807 ts_tablet_manager.cc:1428] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260620 21:53:48.713071 16727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:47495
I20260620 21:53:48.716167 16915 heartbeater.cc:503] Master 127.0.193.254:36223 was elected leader, sending a full tablet report...
Jun 20 21:53:48 dist-test-slave-kk5c krb5kdc[16146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for kudu/127.0.193.193@KRBTEST.COM
I20260620 21:53:48.724431 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:48.712617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.0.193.193:35883)
0620 21:53:48.713324 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:48.713336 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:48.713362 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:53:48.713568 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:48.713574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:48.713788 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:48.713921 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:48.714670 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:48.715489 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:48.716419 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:48.717085 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:48.717218 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:48.719982 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:48.719990 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:48.720000 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:48.720025 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:48.721793 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:48.722556 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:48.722562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:48.722563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:48.722659 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:48.723022 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:48.723025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:48.723026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:48.723382 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:48.723531 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:48.724062 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 21:53:48.724245 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":111,"threads_started":1}
I20260620 21:53:48.724431 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:48.712513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:35883 (local address 127.0.0.1:36326)
0620 21:53:48.713134 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:48.713186 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:48.713397 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:48.713967 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:48.713976 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:48.714412 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:48.714528 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:48.714536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:48.715606 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:48.715611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:48.716279 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:48.716292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:48.717228 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:48.717231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:48.717369 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:48.718054 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:48.718069 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:48.719797 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:48.721909 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:48.721919 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:48.721925 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:48.722408 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:48.722762 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:48.722765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:48.722767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:48.722863 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:48.723534 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:48.723542 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:48.723780 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 21:53:48.723972 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 21:53:48.724236 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":104,"threads_started":1}
I20260620 21:53:48.725970 16849 tablet_service.cc:1514] Processing DeleteTablet for tablet e4195f8eef9f44aab87dc4e41812f726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:28 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:36326
I20260620 21:53:48.726922 16924 tablet_replica.cc:330] stopping tablet replica
I20260620 21:53:48.727349 16924 raft_consensus.cc:2229] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:53:48.727628 16924 raft_consensus.cc:2258] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:53:48.728647 16924 ts_tablet_manager.cc:1905] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:53:48.732443 16924 ts_tablet_manager.cc:1918] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:53:48.732532 16924 log.cc:1196] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal/wals/e4195f8eef9f44aab87dc4e41812f726
I20260620 21:53:48.732808 16924 ts_tablet_manager.cc:1939] T e4195f8eef9f44aab87dc4e41812f726 P 53cd1d8a03dd401b913ec3b6d0fad6c8: Deleting consensus metadata
I20260620 21:53:48.735285 16712 catalog_manager.cc:4841] TS 53cd1d8a03dd401b913ec3b6d0fad6c8 (127.0.193.193:35883): tablet e4195f8eef9f44aab87dc4e41812f726 (table test-table [id=62ba140b47e94f85bb04206fabe27ee1]) successfully deleted
W20260620 21:53:48.816835 16920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:48.817153 16920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:48.817199 16920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:48.821521 16920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:48.821596 16920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:48.821660 16920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:53:48.827782 16920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:33481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=42321
--tserver_master_addrs=127.0.193.254:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:48.828897 16920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:48.829912 16920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:48.840754 16930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:48.840758 16929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:48.840783 16932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:48.841100 16920 server_base.cc:1034] running on GCE node
I20260620 21:53:48.841674 16920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:48.842379 16920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:48.843583 16920 hybrid_clock.cc:648] HybridClock initialized: now 1781992428843547 us; error 58 us; skew 500 ppm
Jun 20 21:53:48 dist-test-slave-kk5c krb5kdc[16146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:48.847671 16920 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:53:48.848826 16920 webserver.cc:460] Webserver started at http://127.0.193.194:42321/ using document root <none> and password file <none>
I20260620 21:53:48.849476 16920 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:48.849558 16920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:48.853803 16920 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 21:53:48.856384 16939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:48.858156 16920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:48.858333 16920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "dabb641223ee4852bfb2b7e67242997a"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "9d54f5a37d68984040059dbbf9a94190"
server_key_iv: "1b5e00122e3774fd5ac6fdd3278359dc"
server_key_version: "encryptionkey@0"
I20260620 21:53:48.859338 16920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:48.872130 16920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:48.875722 16920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:48.875947 16920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:48.876606 16920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:48.877499 16920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:53:48.877583 16920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:48.877678 16920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:53:48.877729 16920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:48.888634 16920 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:33481
I20260620 21:53:48.888741 17052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:33481 every 8 connection(s)
I20260620 21:53:48.889925 16920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:53:48.900265 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 16920
Jun 20 21:53:48 dist-test-slave-kk5c krb5kdc[16146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:48.900832 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:46345
--local_ip_for_outbound_sockets=127.0.193.195
--tserver_master_addrs=127.0.193.254:36223
--webserver_port=33775
--webserver_interface=127.0.193.195
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:53:48.906072 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:48.892191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:38991 (local address 127.0.193.254:36223)
0620 21:53:48.892339 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:48.892342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:48.893725 (+ 1383us) server_negotiation.cc:407] Connection header received
0620 21:53:48.894726 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:48.894730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:48.894791 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:48.894876 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:48.896243 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:48.896818 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:48.898028 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:48.898277 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:48.898402 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:48.901359 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:48.901363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:48.901366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:48.901403 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:48.903512 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:48.904243 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:48.904246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:48.904248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:48.904420 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:48.904841 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:48.904845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:48.904847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:48.905056 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:48.905168 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:48.905734 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 21:53:48.905884 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:53:48.906924 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:48.892539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:38991)
0620 21:53:48.893522 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:48.893567 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:48.894490 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:48.895090 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:48.895097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:48.895561 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:48.896075 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:48.896090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:48.896978 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:48.896981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:48.897886 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:48.897897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:48.898473 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:48.898478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:48.898658 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:48.899445 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:48.899465 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:48.901158 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:48.903678 (+ 2520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:48.903684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:48.903703 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:48.904106 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:48.904549 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:48.904553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:48.904555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:48.904718 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:48.905168 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:48.905177 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:48.905482 (+ 305us) client_negotiation.cc:769] Sending connection context
0620 21:53:48.905694 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 21:53:48.906065 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":605,"threads_started":1}
I20260620 21:53:48.908375 17053 heartbeater.cc:347] Connected to a master server at 127.0.193.254:36223
I20260620 21:53:48.908675 17053 heartbeater.cc:464] Registering TS with master...
I20260620 21:53:48.909204 17053 heartbeater.cc:511] Master 127.0.193.254:36223 requested a full tablet report, sending...
I20260620 21:53:48.910327 16727 ts_manager.cc:194] Registered new tserver with Master: dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481)
I20260620 21:53:48.910982 16727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:38991
W20260620 21:53:49.021827 17056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:53:49.022178 17056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:53:49.022264 17056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:53:49.026465 17056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:53:49.026572 17056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:53:49.026666 17056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:53:49.032668 17056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:46345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=33775
--tserver_master_addrs=127.0.193.254:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:53:49.033861 17056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:53:49.034873 17056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:53:49.045521 17061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:53:49.045516 17064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:49.045629 17056 server_base.cc:1034] running on GCE node
W20260620 21:53:49.045516 17062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:53:49.046218 17056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:53:49.046887 17056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:53:49.048079 17056 hybrid_clock.cc:648] HybridClock initialized: now 1781992429048064 us; error 64 us; skew 500 ppm
Jun 20 21:53:49 dist-test-slave-kk5c krb5kdc[16146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:53:49.051115 17056 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:53:49.052167 17056 webserver.cc:460] Webserver started at http://127.0.193.195:33775/ using document root <none> and password file <none>
I20260620 21:53:49.052771 17056 fs_manager.cc:359] Metadata directory not provided
I20260620 21:53:49.052886 17056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:53:49.057001 17056 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:53:49.059249 17071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:49.060699 17056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:49.060858 17056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "78b7ec3c91454fb08fec7e1824118efc"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "e37955098c20cdf396ae88ea47461d6e"
server_key_iv: "d5a98f232339396bb9445b9c75da21ae"
server_key_version: "encryptionkey@0"
I20260620 21:53:49.061420 17056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:53:49.074626 17056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:53:49.077812 17056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:53:49.078042 17056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:53:49.078676 17056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:53:49.079578 17056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:53:49.079653 17056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:49.079730 17056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:53:49.079774 17056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:53:49.089089 17056 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:46345
I20260620 21:53:49.089155 17184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:46345 every 8 connection(s)
I20260620 21:53:49.090171 17056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:53:49.099152 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 17056
Jun 20 21:53:49 dist-test-slave-kk5c krb5kdc[16146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:53:49.105032 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.092175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:46821 (local address 127.0.193.254:36223)
0620 21:53:49.092318 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:49.092327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:49.093375 (+ 1048us) server_negotiation.cc:407] Connection header received
0620 21:53:49.094242 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:49.094244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:49.094292 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:49.094370 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:49.095768 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.096357 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.097591 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.097854 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.097944 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:49.100958 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:49.100963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:49.100967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:49.100998 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:49.102649 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:49.103236 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:49.103240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:49.103242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:49.103332 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:49.103680 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:49.103684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:49.103685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:49.103899 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:49.103974 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:49.104679 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 21:53:49.104817 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:53:49.105997 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.092480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:46821)
0620 21:53:49.093209 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:49.093253 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:49.094042 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:49.094544 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:49.094551 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:49.094940 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:53:49.095602 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:49.095618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.096483 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.096486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:49.097450 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:49.097459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.098032 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.098035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:49.098191 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:49.098757 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:53:49.098773 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:53:49.100765 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:53:49.102771 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:49.102779 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:49.102800 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:49.103109 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:49.103426 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:53:49.103429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:53:49.103430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:53:49.103582 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:53:49.103999 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:53:49.104012 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:53:49.104353 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 21:53:49.104578 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 21:53:49.104902 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":125,"threads_started":1}
I20260620 21:53:49.107421 17185 heartbeater.cc:347] Connected to a master server at 127.0.193.254:36223
I20260620 21:53:49.107694 17185 heartbeater.cc:464] Registering TS with master...
I20260620 21:53:49.108394 17185 heartbeater.cc:511] Master 127.0.193.254:36223 requested a full tablet report, sending...
I20260620 21:53:49.109354 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.099970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.0.193.254:36223)
0620 21:53:49.100246 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:49.100251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:49.100442 (+ 191us) server_negotiation.cc:407] Connection header received
0620 21:53:49.100576 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:49.100579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:49.100631 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:49.100737 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:49.102165 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.102728 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.103780 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.104077 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.104177 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:49.105635 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:49.105638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:49.105641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:49.105666 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:49.107403 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:49.107911 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:49.107913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:49.107915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:49.107987 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:49.108316 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:49.108319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:49.108320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:49.108524 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:49.108695 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:49.108958 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 21:53:49.109156 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
I20260620 21:53:49.109659 16727 ts_manager.cc:194] Registered new tserver with Master: 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345)
I20260620 21:53:49.110329 16727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:46821
I20260620 21:53:49.111392 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:53:49.115720 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.112107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.0.193.254:36223)
0620 21:53:49.112242 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:49.112246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:49.112470 (+ 224us) server_negotiation.cc:407] Connection header received
0620 21:53:49.112610 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:49.112612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:49.112698 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:49.112783 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:53:49.112977 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.113460 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.114349 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.114612 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.114702 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:49.115147 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:53:49.115393 (+ 246us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 21:53:49.115618 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52562: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 21:53:49.115818 17188 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52562: token expired
W20260620 21:53:49.116204 16654 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.193.254:36223)
I20260620 21:53:49.124156 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.116594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.0.193.254:36223)
0620 21:53:49.116727 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:49.116730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:49.116782 (+ 52us) server_negotiation.cc:407] Connection header received
0620 21:53:49.116945 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:49.116947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:49.116985 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:49.117100 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:53:49.117673 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.118172 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.119086 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.119377 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.119460 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:49.120751 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:53:49.120754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:53:49.120756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:53:49.120779 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:53:49.122133 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:49.122709 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:49.122712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:49.122713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:49.122794 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:53:49.123177 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:53:49.123180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:53:49.123242 (+ 62us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:53:49.123470 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:53:49.123586 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:49.123837 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 21:53:49.123943 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:53:49.126715 16726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:53:49.130307 16726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:53:49.138674 16849 tablet_service.cc:1467] Processing CreateTablet for tablet e53726a749944c0daa3ddb799c4b4c2f (DEFAULT_TABLE table=test-table [id=efaa38631f204cca875da4553278c0a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:53:49.139392 16849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53726a749944c0daa3ddb799c4b4c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:49.142379 16807 tablet_bootstrap.cc:492] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8: Bootstrap starting.
I20260620 21:53:49.143447 16807 tablet_bootstrap.cc:654] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:49.143544 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.137251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:33481 (local address 127.0.0.1:56850)
0620 21:53:49.137554 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:49.137574 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:49.137658 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:49.138748 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:49.138750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:49.138767 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:49.138906 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:49.138915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.140603 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.140608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:49.141788 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:49.141819 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.142980 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.142983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:49.143101 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:49.143153 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 21:53:49.143253 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 21:53:49.143355 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":28}
I20260620 21:53:49.144028 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.137827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.0.193.194:33481)
0620 21:53:49.138239 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:49.138252 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:49.138275 (+ 23us) server_negotiation.cc:407] Connection header received
0620 21:53:49.138355 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:49.138360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:49.138583 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:49.138752 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:49.139069 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.140444 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.141951 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.142857 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.142979 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:49.143155 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:49.143824 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 21:53:49.143914 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":108,"threads_started":1}
I20260620 21:53:49.144532 17194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.137467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:46345 (local address 127.0.0.1:44432)
0620 21:53:49.137763 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:49.137775 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:49.137902 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:49.139728 (+ 1826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:49.139731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:49.139747 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:49.139862 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:49.139868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.141252 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.141256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:49.142678 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:49.142690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.144060 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.144064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:49.144172 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:49.144198 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:53:49.144287 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 21:53:49.144378 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":4,"thread_start_us":113,"threads_started":1}
I20260620 21:53:49.144578 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.138025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44432 (local address 127.0.193.195:46345)
0620 21:53:49.139102 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:49.139114 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:49.139136 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:53:49.139225 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:49.139230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:49.139408 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:49.139563 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:49.139994 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.141108 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.142806 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.143677 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.143803 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:49.143967 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:49.144337 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 21:53:49.144437 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":141,"threads_started":1}
I20260620 21:53:49.145527 16807 tablet_bootstrap.cc:492] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8: No bootstrap required, opened a new log
I20260620 21:53:49.145655 16807 ts_tablet_manager.cc:1397] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260620 21:53:49.146186 16807 raft_consensus.cc:348] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:49.146332 16807 raft_consensus.cc:374] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:49.146370 16807 raft_consensus.cc:729] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53cd1d8a03dd401b913ec3b6d0fad6c8, State: Initialized, Role: FOLLOWER
I20260620 21:53:49.146502 16807 consensus_queue.cc:260] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:49.147393 16807 ts_tablet_manager.cc:1428] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 21:53:49.147140 16987 tablet_service.cc:1467] Processing CreateTablet for tablet e53726a749944c0daa3ddb799c4b4c2f (DEFAULT_TABLE table=test-table [id=efaa38631f204cca875da4553278c0a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:53:49.148443 16987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53726a749944c0daa3ddb799c4b4c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:49.148186 17119 tablet_service.cc:1467] Processing CreateTablet for tablet e53726a749944c0daa3ddb799c4b4c2f (DEFAULT_TABLE table=test-table [id=efaa38631f204cca875da4553278c0a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:53:49.149158 17119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53726a749944c0daa3ddb799c4b4c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:53:49.156011 17199 tablet_bootstrap.cc:492] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc: Bootstrap starting.
I20260620 21:53:49.156023 17198 tablet_bootstrap.cc:492] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: Bootstrap starting.
I20260620 21:53:49.158401 17198 tablet_bootstrap.cc:654] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:49.158411 17199 tablet_bootstrap.cc:654] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc: Neither blocks nor log segments found. Creating new log.
I20260620 21:53:49.159157 17199 log.cc:824] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:49.159157 17198 log.cc:824] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:53:49.161166 17198 tablet_bootstrap.cc:492] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: No bootstrap required, opened a new log
I20260620 21:53:49.161383 17198 ts_tablet_manager.cc:1397] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:53:49.161473 17199 tablet_bootstrap.cc:492] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc: No bootstrap required, opened a new log
I20260620 21:53:49.161672 17199 ts_tablet_manager.cc:1397] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:53:49.164757 17198 raft_consensus.cc:348] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:49.165091 17198 raft_consensus.cc:374] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:49.165180 17198 raft_consensus.cc:729] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabb641223ee4852bfb2b7e67242997a, State: Initialized, Role: FOLLOWER
I20260620 21:53:49.165076 17199 raft_consensus.cc:348] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:49.165457 17199 raft_consensus.cc:374] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:53:49.165549 17199 raft_consensus.cc:729] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b7ec3c91454fb08fec7e1824118efc, State: Initialized, Role: FOLLOWER
I20260620 21:53:49.165764 17198 consensus_queue.cc:260] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:49.165961 17199 consensus_queue.cc:260] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:49.166577 17053 heartbeater.cc:503] Master 127.0.193.254:36223 was elected leader, sending a full tablet report...
I20260620 21:53:49.166678 17185 heartbeater.cc:503] Master 127.0.193.254:36223 was elected leader, sending a full tablet report...
I20260620 21:53:49.166936 17198 ts_tablet_manager.cc:1428] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 21:53:49.167949 17199 ts_tablet_manager.cc:1428] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
W20260620 21:53:49.183674 16916 tablet.cc:2369] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:53:49.272225 16921 raft_consensus.cc:482] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:53:49.272477 16921 raft_consensus.cc:504] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:49.273685 16921 leader_election.cc:288] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481), 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345)
I20260620 21:53:49.278095 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.274671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:33809 (local address 127.0.193.195:46345)
0620 21:53:49.274812 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:49.274819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:49.274868 (+ 49us) server_negotiation.cc:407] Connection header received
0620 21:53:49.275018 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:49.275021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:49.275069 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:49.275155 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:49.275483 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.275998 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.277113 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.277527 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.277660 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:49.277714 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:49.277900 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 21:53:49.277970 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:53:49.278120 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.274493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:46345 (local address 127.0.193.193:33809)
0620 21:53:49.274784 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:49.274795 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:49.274903 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:49.275171 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:49.275173 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:49.275238 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:49.275361 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:49.275366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.276130 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.276134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:49.276979 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:49.276988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.277651 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.277655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:49.277761 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:49.277797 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 21:53:49.277885 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 21:53:49.277961 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":41,"spinlock_wait_cycles":20608,"thread_start_us":83,"threads_started":1}
I20260620 21:53:49.278249 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.274158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:49035 (local address 127.0.193.194:33481)
0620 21:53:49.274305 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:49.274309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:49.274520 (+ 211us) server_negotiation.cc:407] Connection header received
0620 21:53:49.274767 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:49.274771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:49.274832 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:49.275002 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:53:49.275520 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.276052 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.277178 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.277631 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.277769 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:49.277821 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:49.278025 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 21:53:49.278099 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:53:49.278582 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.274058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:33481 (local address 127.0.193.193:49035)
0620 21:53:49.274415 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:49.274433 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:53:49.274565 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:53:49.275173 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:53:49.275176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:53:49.275250 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:53:49.275360 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:49.275366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.276171 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.276178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:49.277037 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:53:49.277046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.277769 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.277774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:53:49.277876 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:49.277910 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 21:53:49.278007 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 21:53:49.278084 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":2,"spinlock_wait_cycles":19712,"thread_start_us":100,"threads_started":1}
I20260620 21:53:49.279049 17139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53726a749944c0daa3ddb799c4b4c2f" candidate_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b7ec3c91454fb08fec7e1824118efc" is_pre_election: true
I20260620 21:53:49.279294 17007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53726a749944c0daa3ddb799c4b4c2f" candidate_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabb641223ee4852bfb2b7e67242997a" is_pre_election: true
I20260620 21:53:49.279422 17139 raft_consensus.cc:2454] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53cd1d8a03dd401b913ec3b6d0fad6c8 in term 0.
I20260620 21:53:49.279552 17007 raft_consensus.cc:2454] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53cd1d8a03dd401b913ec3b6d0fad6c8 in term 0.
I20260620 21:53:49.279964 16803 leader_election.cc:302] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53cd1d8a03dd401b913ec3b6d0fad6c8, 78b7ec3c91454fb08fec7e1824118efc; no voters:
I20260620 21:53:49.280376 16921 raft_consensus.cc:2781] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:53:49.280452 16921 raft_consensus.cc:482] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:53:49.280483 16921 raft_consensus.cc:3037] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:49.281510 16921 raft_consensus.cc:504] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:49.281879 16921 leader_election.cc:288] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 1 election: Requested vote from peers dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481), 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345)
I20260620 21:53:49.282378 17007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53726a749944c0daa3ddb799c4b4c2f" candidate_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabb641223ee4852bfb2b7e67242997a"
I20260620 21:53:49.282490 17007 raft_consensus.cc:3037] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:49.282506 17139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53726a749944c0daa3ddb799c4b4c2f" candidate_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b7ec3c91454fb08fec7e1824118efc"
I20260620 21:53:49.282665 17139 raft_consensus.cc:3037] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:53:49.283685 17007 raft_consensus.cc:2454] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53cd1d8a03dd401b913ec3b6d0fad6c8 in term 1.
I20260620 21:53:49.283924 17139 raft_consensus.cc:2454] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53cd1d8a03dd401b913ec3b6d0fad6c8 in term 1.
I20260620 21:53:49.284053 16803 leader_election.cc:302] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53cd1d8a03dd401b913ec3b6d0fad6c8, dabb641223ee4852bfb2b7e67242997a; no voters:
I20260620 21:53:49.284373 16921 raft_consensus.cc:2781] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:53:49.284529 16921 raft_consensus.cc:686] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 LEADER]: Becoming Leader. State: Replica: 53cd1d8a03dd401b913ec3b6d0fad6c8, State: Running, Role: LEADER
I20260620 21:53:49.284785 16921 consensus_queue.cc:237] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:53:49.287767 16727 catalog_manager.cc:5495] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53cd1d8a03dd401b913ec3b6d0fad6c8 (127.0.193.193). New cstate: current_term: 1 leader_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } health_report { overall_health: HEALTHY } } }
I20260620 21:53:49.299180 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.295164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.0.193.193:35883)
0620 21:53:49.295541 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 21:53:49.295549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:53:49.295561 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:53:49.295664 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:53:49.295670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:53:49.295745 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:53:49.295882 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:53:49.296127 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.296666 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.297546 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:53:49.297800 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:53:49.297968 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:53:49.298216 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:53:49.298706 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:53:49.298810 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 21:53:49.298899 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 21:53:49.298952 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":120,"threads_started":1}
I20260620 21:53:49.303856 17007 raft_consensus.cc:1261] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a [term 1 FOLLOWER]: Refusing update from remote peer 53cd1d8a03dd401b913ec3b6d0fad6c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:53:49.303855 17139 raft_consensus.cc:1261] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc [term 1 FOLLOWER]: Refusing update from remote peer 53cd1d8a03dd401b913ec3b6d0fad6c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:53:49.304658 16921 consensus_queue.cc:1035] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:53:49.304919 17206 consensus_queue.cc:1035] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:53:49.325536 16727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:53:49.325700 16727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:53:49.328284 16727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Sending DeleteTablet for 3 replicas of tablet e53726a749944c0daa3ddb799c4b4c2f
I20260620 21:53:49.329177 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 16765
I20260620 21:53:49.329185 16987 tablet_service.cc:1514] Processing DeleteTablet for tablet e53726a749944c0daa3ddb799c4b4c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:49 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:56850
I20260620 21:53:49.329181 17119 tablet_service.cc:1514] Processing DeleteTablet for tablet e53726a749944c0daa3ddb799c4b4c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:53:49 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:44432
I20260620 21:53:49.331146 17220 tablet_replica.cc:330] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc: stopping tablet replica
I20260620 21:53:49.331540 17220 raft_consensus.cc:2229] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:53:49.331929 17220 raft_consensus.cc:2258] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:53:49.332348 17219 tablet_replica.cc:330] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: stopping tablet replica
I20260620 21:53:49.332647 17219 raft_consensus.cc:2229] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:53:49.333263 17219 raft_consensus.cc:2258] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:53:49.333859 17220 ts_tablet_manager.cc:1905] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:53:49.334731 17219 ts_tablet_manager.cc:1905] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 21:53:49.337044 16712 connection.cc:535] client connection to 127.0.193.193:35883 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 21:53:49.337240 16712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 21:53:49.337280 17220 ts_tablet_manager.cc:1918] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:53:49.337378 17220 log.cc:1196] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/e53726a749944c0daa3ddb799c4b4c2f
I20260620 21:53:49.337392 17219 ts_tablet_manager.cc:1918] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:53:49.337483 17219 log.cc:1196] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/e53726a749944c0daa3ddb799c4b4c2f
I20260620 21:53:49.337539 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 16920
I20260620 21:53:49.337772 17220 ts_tablet_manager.cc:1939] T e53726a749944c0daa3ddb799c4b4c2f P 78b7ec3c91454fb08fec7e1824118efc: Deleting consensus metadata
I20260620 21:53:49.337877 17219 ts_tablet_manager.cc:1939] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: Deleting consensus metadata
I20260620 21:53:49.338838 17194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.338423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:35883 (local address 127.0.0.1:36346)
0620 21:53:49.338699 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:49.338759 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260620 21:53:49.339236 16713 catalog_manager.cc:4841] TS 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345): tablet e53726a749944c0daa3ddb799c4b4c2f (table test-table [id=efaa38631f204cca875da4553278c0a1]) successfully deleted
W20260620 21:53:49.339560 16712 catalog_manager.cc:4584] TS 53cd1d8a03dd401b913ec3b6d0fad6c8 (127.0.193.193:35883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e53726a749944c0daa3ddb799c4b4c2f: Network error: Client connection negotiation failed: client connection to 127.0.193.193:35883: connect: Connection refused (error 111)
I20260620 21:53:49.344839 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 17056
I20260620 21:53:49.346692 17194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:53:49.345184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:33481 (local address 127.0.0.1:56864)
0620 21:53:49.346484 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0620 21:53:49.346571 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.194:33481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1181}
I20260620 21:53:49.350863 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 16694
I20260620 21:54:09.359709 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36223
--webserver_interface=127.0.193.254
--webserver_port=33287
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:36223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:09.475070 17222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:09.475425 17222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:09.475468 17222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:09.479734 17222 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 21:54:09.479815 17222 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 21:54:09.479830 17222 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 21:54:09.479844 17222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:09.479864 17222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:09.479883 17222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:09.479900 17222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:09.479915 17222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:09.485498 17222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:36223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=33287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:09.486689 17222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:09.487717 17222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:09.496964 17227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:09.497045 17222 server_base.cc:1034] running on GCE node
W20260620 21:54:09.496974 17228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:09.496974 17230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:09.498029 17222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:09.499159 17222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:09.500415 17222 hybrid_clock.cc:648] HybridClock initialized: now 1781992449500385 us; error 47 us; skew 500 ppm
Jun 20 21:54:09 dist-test-slave-kk5c krb5kdc[16146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:09.507117 17222 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:54:09.508445 17222 webserver.cc:460] Webserver started at http://127.0.193.254:33287/ using document root <none> and password file <none>
I20260620 21:54:09.509117 17222 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:09.509224 17222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:09.513847 17222 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:54:09.516567 17237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:09.518178 17222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:09.518368 17222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "033df915270e41eb8eb89bf91e36ccdf"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "6094e28a6174d5b75e91737caca41bd1"
server_key_iv: "3c07d838668a8c11944abe0f5581700c"
server_key_version: "encryptionkey@0"
I20260620 21:54:09.519002 17222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:09.535459 17222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:09.539253 17222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:09.539491 17222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:09.547561 17222 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:36223
I20260620 21:54:09.547855 17289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:36223 every 8 connection(s)
I20260620 21:54:09.548781 17222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:09.553627 17290 sys_catalog.cc:263] Verifying existing consensus state
I20260620 21:54:09.555583 17290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Bootstrap starting.
I20260620 21:54:09.557435 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 17222
I20260620 21:54:09.557962 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:35883
--local_ip_for_outbound_sockets=127.0.193.193
--tserver_master_addrs=127.0.193.254:36223
--webserver_port=38967
--webserver_interface=127.0.193.193
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:54:09.574409 17290 log.cc:824] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:09.586287 17290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:54:09.586759 17290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Bootstrap complete.
I20260620 21:54:09.590973 17290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } }
I20260620 21:54:09.591470 17290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 033df915270e41eb8eb89bf91e36ccdf, State: Initialized, Role: FOLLOWER
I20260620 21:54:09.591966 17290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } }
I20260620 21:54:09.592114 17290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:09.592203 17290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:09.592341 17290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 2 FOLLOWER]: Advancing to term 3
I20260620 21:54:09.594321 17290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } }
I20260620 21:54:09.594677 17290 leader_election.cc:302] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 033df915270e41eb8eb89bf91e36ccdf; no voters:
I20260620 21:54:09.595058 17290 leader_election.cc:288] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 21:54:09.595151 17294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 3 FOLLOWER]: Leader election won for term 3
I20260620 21:54:09.595386 17294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [term 3 LEADER]: Becoming Leader. State: Replica: 033df915270e41eb8eb89bf91e36ccdf, State: Running, Role: LEADER
I20260620 21:54:09.595743 17294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } }
I20260620 21:54:09.596355 17290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:09.596696 17294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 033df915270e41eb8eb89bf91e36ccdf. Latest consensus state: current_term: 3 leader_uuid: "033df915270e41eb8eb89bf91e36ccdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } } }
I20260620 21:54:09.597507 17294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:09.597385 17295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "033df915270e41eb8eb89bf91e36ccdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033df915270e41eb8eb89bf91e36ccdf" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36223 } } }
I20260620 21:54:09.598233 17295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:09.598490 17303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:09.604518 17303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:09.605527 17303 catalog_manager.cc:1259] Loaded cluster ID: 221ac7f7948a4808935438b1d3425324
I20260620 21:54:09.605618 17303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:09.608160 17303 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:09.608976 17303 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Loaded TSK: 0
I20260620 21:54:09.609089 17303 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 033df915270e41eb8eb89bf91e36ccdf: Loaded TSK: 1
F20260620 21:54:09.609838 17303 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f51eb5509ad google::LogMessage::Fail() at ??:0
@ 0x7f51eb55388a google::LogMessage::SendToLog() at ??:0
@ 0x7f51eb5504de google::LogMessage::Flush() at ??:0
@ 0x7f51eb551a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f51ee21418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f51ee20cf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f51f5eb57bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f51f5e8f99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f51f5e90178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f51f5ed1df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f51f5ef082b std::function<>::operator()() at ??:0
@ 0x7f51f5e8fb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f51f5e90f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f51f5e8d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f51f5ed1640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f51ed7a23d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f51ed7a2c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f51ed7a43f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f51ed793512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f51eedd76db start_thread at ??:0
@ 0x7f51ea04e61f clone at ??:0
W20260620 21:54:09.700860 17292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:09.701174 17292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:09.701223 17292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:09.706223 17292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:09.706277 17292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:09.706342 17292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:54:09.712687 17292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:35883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=38967
--tserver_master_addrs=127.0.193.254:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:09.713807 17292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:09.715010 17292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:54:09.726003 17292 server_base.cc:1034] running on GCE node
W20260620 21:54:09.725961 17338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:09.726187 17340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:09.725961 17337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:09.726694 17292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:09.727523 17292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:09.728708 17292 hybrid_clock.cc:648] HybridClock initialized: now 1781992449728691 us; error 37 us; skew 500 ppm
Jun 20 21:54:09 dist-test-slave-kk5c krb5kdc[16146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:09.731521 17292 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:54:09.732522 17292 webserver.cc:460] Webserver started at http://127.0.193.193:38967/ using document root <none> and password file <none>
I20260620 21:54:09.733111 17292 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:09.733173 17292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:09.737370 17292 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:54:09.739835 17347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:09.741334 17292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:09.741513 17292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "cc817a1287f71bce6b3104fb4fac0e46"
server_key_iv: "313e310b64f947cf0864fbbd6b4f826f"
server_key_version: "encryptionkey@0"
I20260620 21:54:09.742156 17292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:09.774461 17292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:09.778043 17292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:09.778263 17292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:09.778939 17292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:09.780197 17354 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 21:54:09.784446 17292 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 21:54:09.784519 17292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 21:54:09.784574 17292 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 21:54:09.786331 17292 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 21:54:09.786397 17292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 21:54:09.786661 17354 tablet_bootstrap.cc:492] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8: Bootstrap starting.
I20260620 21:54:09.797289 17292 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:35883
I20260620 21:54:09.797564 17461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:35883 every 8 connection(s)
I20260620 21:54:09.798734 17292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:54:09.806152 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 17292
I20260620 21:54:09.806672 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:33481
--local_ip_for_outbound_sockets=127.0.193.194
--tserver_master_addrs=127.0.193.254:36223
--webserver_port=42321
--webserver_interface=127.0.193.194
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:54:09.807955 17354 log.cc:824] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:09.818354 17354 tablet_bootstrap.cc:492] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 21:54:09.818898 17354 tablet_bootstrap.cc:492] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8: Bootstrap complete.
I20260620 21:54:09.820524 17354 ts_tablet_manager.cc:1397] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.015s
I20260620 21:54:09.823926 17354 raft_consensus.cc:348] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:54:09.824481 17354 raft_consensus.cc:729] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53cd1d8a03dd401b913ec3b6d0fad6c8, State: Initialized, Role: FOLLOWER
I20260620 21:54:09.825047 17354 consensus_queue.cc:260] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:54:09.826138 17354 ts_tablet_manager.cc:1428] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260620 21:54:09.860157 17464 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:54:09.801265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:51909)
0620 21:54:09.801836 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:09.801895 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:09.803015 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:09.859321 (+ 56306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":139,"threads_started":1}
W20260620 21:54:09.860780 17462 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:36223 (0 consecutive failures): Network error: Failed to ping master at 127.0.193.254:36223: Client connection negotiation failed: client connection to 127.0.193.254:36223: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 21:54:09.861694 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:09.861406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:50429)
0620 21:54:09.861551 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:09.861632 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 21:54:09.862421 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:09.862202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:52141)
0620 21:54:09.862313 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:09.862363 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 21:54:09.862653 17462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:54:09.933566 17465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:09.933858 17465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:09.933904 17465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:09.938369 17465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:09.938446 17465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:09.938539 17465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:54:09.944188 17465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:33481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=42321
--tserver_master_addrs=127.0.193.254:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:09.945309 17465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:09.946367 17465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:54:09.956903 17465 server_base.cc:1034] running on GCE node
W20260620 21:54:09.956857 17475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:09.956799 17472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:09.956842 17473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:09.957538 17465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:09.958268 17465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:09.959504 17465 hybrid_clock.cc:648] HybridClock initialized: now 1781992449959493 us; error 31 us; skew 500 ppm
Jun 20 21:54:09 dist-test-slave-kk5c krb5kdc[16146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:09.962728 17465 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:54:09.963762 17465 webserver.cc:460] Webserver started at http://127.0.193.194:42321/ using document root <none> and password file <none>
I20260620 21:54:09.964367 17465 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:09.964464 17465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:09.968850 17465 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:54:09.971519 17482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:09.972952 17465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:09.973145 17465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "dabb641223ee4852bfb2b7e67242997a"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "9d54f5a37d68984040059dbbf9a94190"
server_key_iv: "1b5e00122e3774fd5ac6fdd3278359dc"
server_key_version: "encryptionkey@0"
I20260620 21:54:09.973802 17465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:09.986133 17465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:09.989673 17465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:09.989907 17465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:09.990526 17465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:09.991711 17489 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 21:54:09.995491 17489 ts_tablet_manager.cc:1852] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 21:54:09.995934 17489 ts_tablet_manager.cc:1905] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:54:10.000334 17489 ts_tablet_manager.cc:1918] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:54:10.000582 17489 ts_tablet_manager.cc:1939] T e53726a749944c0daa3ddb799c4b4c2f P dabb641223ee4852bfb2b7e67242997a: Deleting consensus metadata
I20260620 21:54:10.001313 17465 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 21:54:10.001387 17465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260620 21:54:10.001451 17465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:10.001523 17465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:10.011986 17465 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:33481
I20260620 21:54:10.012013 17596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:33481 every 8 connection(s)
I20260620 21:54:10.012962 17465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:54:10.016470 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 17465
I20260620 21:54:10.016952 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:46345
--local_ip_for_outbound_sockets=127.0.193.195
--tserver_master_addrs=127.0.193.254:36223
--webserver_port=33775
--webserver_interface=127.0.193.195
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:54:10.016955 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:10.015409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:56669)
0620 21:54:10.015966 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:10.016082 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":130,"threads_started":1}
W20260620 21:54:10.017741 17597 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:36223 (0 consecutive failures): Network error: Failed to ping master at 127.0.193.254:36223: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
I20260620 21:54:10.018723 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:10.018403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:49741)
0620 21:54:10.018596 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:10.018657 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 21:54:10.019673 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:10.019341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:45085)
0620 21:54:10.019509 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:10.019584 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 21:54:10.019891 17597 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 21:54:10.137768 17600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:10.138046 17600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:10.138092 17600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:10.142246 17600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:10.142315 17600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:10.142375 17600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:54:10.147939 17600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:46345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=33775
--tserver_master_addrs=127.0.193.254:36223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:10.148981 17600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:10.150023 17600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:10.160267 17608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:10.160318 17606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:10.160380 17600 server_base.cc:1034] running on GCE node
W20260620 21:54:10.160269 17605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:10.160992 17600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:10.161680 17600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:10.162851 17600 hybrid_clock.cc:648] HybridClock initialized: now 1781992450162817 us; error 46 us; skew 500 ppm
Jun 20 21:54:10 dist-test-slave-kk5c krb5kdc[16146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:10.165735 17600 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:54:10.166601 17600 webserver.cc:460] Webserver started at http://127.0.193.195:33775/ using document root <none> and password file <none>
I20260620 21:54:10.167138 17600 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:10.167273 17600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:10.171139 17600 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:10.173355 17615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:10.174656 17600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:54:10.174762 17600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "78b7ec3c91454fb08fec7e1824118efc"
format_stamp: "Formatted at 2026-06-20 21:53:27 on dist-test-slave-kk5c"
server_key: "e37955098c20cdf396ae88ea47461d6e"
server_key_iv: "d5a98f232339396bb9445b9c75da21ae"
server_key_version: "encryptionkey@0"
I20260620 21:54:10.175384 17600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:10.194517 17600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:10.197839 17600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:10.198052 17600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:10.198668 17600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:10.199581 17600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:10.199630 17600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:10.199671 17600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:10.199728 17600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:10.209398 17600 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:46345
I20260620 21:54:10.209420 17728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:46345 every 8 connection(s)
I20260620 21:54:10.210474 17600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:54:10.215065 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:10.212959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:35601)
0620 21:54:10.213898 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:10.214074 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":139,"threads_started":1}
I20260620 21:54:10.215262 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 17600
W20260620 21:54:10.215623 17729 heartbeater.cc:644] Failed to heartbeat to 127.0.193.254:36223 (0 consecutive failures): Network error: Failed to ping master at 127.0.193.254:36223: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
I20260620 21:54:10.216241 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:10.216005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:42799)
0620 21:54:10.216126 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:10.216183 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 21:54:10.216935 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:10.216720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:45955)
0620 21:54:10.216822 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:10.216871 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 21:54:10.217163 17729 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 21:54:10.863819 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:10.863355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:54291)
0620 21:54:10.863654 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:10.863713 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
I20260620 21:54:11.021000 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:11.020538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:43827)
0620 21:54:11.020823 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:11.020897 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":75,"threads_started":1}
I20260620 21:54:11.218319 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:11.217874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:60299)
0620 21:54:11.218141 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:11.218208 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
I20260620 21:54:11.440179 17736 raft_consensus.cc:482] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:54:11.440387 17736 raft_consensus.cc:504] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:54:11.441391 17736 leader_election.cc:288] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481), 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345)
Jun 20 21:54:11 dist-test-slave-kk5c krb5kdc[16146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.194@KRBTEST.COM
Jun 20 21:54:11 dist-test-slave-kk5c krb5kdc[16146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.195@KRBTEST.COM
I20260620 21:54:11.455942 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:11.442413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:56981 (local address 127.0.193.195:46345)
0620 21:54:11.442741 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:11.442759 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:11.442789 (+ 30us) server_negotiation.cc:407] Connection header received
0620 21:54:11.443422 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:11.443428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:11.443816 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:11.444104 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:11.445007 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:11.446070 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:11.447515 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:11.447998 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:11.448205 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:11.451288 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:11.451293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:11.451303 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:11.451339 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:11.453145 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:11.453769 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:11.453776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:11.453781 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:11.453906 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:11.454288 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:11.454293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:11.454295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:11.454711 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:11.454855 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:11.455478 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 21:54:11.455740 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":128,"threads_started":1}
I20260620 21:54:11.455937 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:11.442039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:46345 (local address 127.0.193.193:56981)
0620 21:54:11.442369 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:11.442384 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:11.442473 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:11.444066 (+ 1593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:11.444069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:11.444359 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:11.444822 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:11.444839 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:11.446247 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:11.446250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:11.447381 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:11.447391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:11.448134 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:11.448137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:11.448373 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:11.449058 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:54:11.449074 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:11.451081 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:11.453269 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:11.453272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:11.453284 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:11.453652 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:11.454011 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:11.454021 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:11.454023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:11.454191 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:11.454855 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:11.454858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:11.455123 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 21:54:11.455364 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 21:54:11.455754 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":200,"spinlock_wait_cycles":60288,"thread_start_us":78,"threads_started":1}
I20260620 21:54:11.455973 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:11.441834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:44035 (local address 127.0.193.194:33481)
0620 21:54:11.442103 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:11.442120 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:11.442166 (+ 46us) server_negotiation.cc:407] Connection header received
0620 21:54:11.442926 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:11.442934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:11.443348 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:11.443600 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:11.445035 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:11.446140 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:11.447514 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:11.448000 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:11.448187 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:11.451029 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:11.451034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:11.451047 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:11.451075 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:11.453084 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:11.453770 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:11.453782 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:11.453786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:11.453903 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:11.454288 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:11.454292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:11.454294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:11.454711 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:11.454855 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:11.455479 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 21:54:11.455772 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":87,"threads_started":1}
I20260620 21:54:11.456043 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:11.441669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:33481 (local address 127.0.193.193:44035)
0620 21:54:11.441913 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:11.441930 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:11.442033 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:11.443745 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:11.443751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:11.444318 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:11.444846 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:11.444856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:11.446281 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:11.446286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:11.447380 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:11.447391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:11.448132 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:11.448136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:11.448380 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:11.449116 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:54:11.449129 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:11.450769 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:11.453264 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:11.453270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:11.453284 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:11.453641 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:11.454010 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:11.454013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:11.454015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:11.454186 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:11.454843 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:11.454849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:11.455124 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 21:54:11.455363 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 21:54:11.455844 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":327,"spinlock_wait_cycles":3072,"thread_start_us":87,"threads_started":1}
W20260620 21:54:11.458540 17350 leader_election.cc:341] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481): Not found: Tablet not found: e53726a749944c0daa3ddb799c4b4c2f
W20260620 21:54:11.458793 17350 leader_election.cc:341] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345): Not found: Tablet not found: e53726a749944c0daa3ddb799c4b4c2f
I20260620 21:54:11.458906 17350 leader_election.cc:302] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53cd1d8a03dd401b913ec3b6d0fad6c8; no voters: 78b7ec3c91454fb08fec7e1824118efc, dabb641223ee4852bfb2b7e67242997a
I20260620 21:54:11.459309 17736 raft_consensus.cc:2726] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:54:11.864984 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:11.864649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:34661)
0620 21:54:11.864811 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:11.864899 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 21:54:12.022318 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:12.021860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:43013)
0620 21:54:12.022148 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:12.022225 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
I20260620 21:54:12.219633 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:12.219153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:54169)
0620 21:54:12.219481 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:12.219540 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":179,"threads_started":1}
I20260620 21:54:12.866333 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:12.865892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:37901)
0620 21:54:12.866169 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:12.866226 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":74,"threads_started":1}
I20260620 21:54:13.023834 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:13.023293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:47849)
0620 21:54:13.023651 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:13.023725 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":88,"threads_started":1}
I20260620 21:54:13.220947 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:13.220439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:57263)
0620 21:54:13.220750 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:13.220812 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":77,"threads_started":1}
I20260620 21:54:13.563560 17746 raft_consensus.cc:482] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:54:13.563699 17746 raft_consensus.cc:504] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:54:13.564088 17746 leader_election.cc:288] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481), 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345)
W20260620 21:54:13.565243 17350 leader_election.cc:341] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481): Not found: Tablet not found: e53726a749944c0daa3ddb799c4b4c2f
W20260620 21:54:13.565526 17350 leader_election.cc:341] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345): Not found: Tablet not found: e53726a749944c0daa3ddb799c4b4c2f
I20260620 21:54:13.565649 17350 leader_election.cc:302] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53cd1d8a03dd401b913ec3b6d0fad6c8; no voters: 78b7ec3c91454fb08fec7e1824118efc, dabb641223ee4852bfb2b7e67242997a
I20260620 21:54:13.565908 17746 raft_consensus.cc:2726] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:54:13.867925 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:13.867396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:43151)
0620 21:54:13.867712 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:13.867785 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":94,"threads_started":1}
I20260620 21:54:14.025107 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:14.024635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:59835)
0620 21:54:14.024925 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:14.024982 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
I20260620 21:54:14.222525 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:14.221980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:60947)
0620 21:54:14.222352 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:14.222428 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":94,"threads_started":1}
I20260620 21:54:14.869707 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:14.869012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:47549)
0620 21:54:14.869369 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:14.869435 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":78,"threads_started":1}
I20260620 21:54:15.026741 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:15.026207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:55761)
0620 21:54:15.026558 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:15.026627 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":139,"threads_started":1}
I20260620 21:54:15.224128 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:15.223615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:59109)
0620 21:54:15.223938 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:15.224016 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":96,"threads_started":1}
I20260620 21:54:15.641682 17753 raft_consensus.cc:482] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:54:15.641847 17753 raft_consensus.cc:504] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:54:15.642357 17753 leader_election.cc:288] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481), 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345)
W20260620 21:54:15.643405 17350 leader_election.cc:341] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481): Not found: Tablet not found: e53726a749944c0daa3ddb799c4b4c2f
W20260620 21:54:15.643528 17350 leader_election.cc:341] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345): Not found: Tablet not found: e53726a749944c0daa3ddb799c4b4c2f
I20260620 21:54:15.643612 17350 leader_election.cc:302] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53cd1d8a03dd401b913ec3b6d0fad6c8; no voters: 78b7ec3c91454fb08fec7e1824118efc, dabb641223ee4852bfb2b7e67242997a
I20260620 21:54:15.643814 17753 raft_consensus.cc:2726] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:54:15.871538 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:15.870920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:37731)
0620 21:54:15.871364 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:15.871435 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":85,"threads_started":1}
I20260620 21:54:16.028208 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:16.027738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:34539)
0620 21:54:16.027998 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:16.028062 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260620 21:54:16.225458 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:16.225005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:55667)
0620 21:54:16.225302 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:16.225363 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
I20260620 21:54:16.873047 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:16.872573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:38071)
0620 21:54:16.872886 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:16.872949 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":158,"threads_started":1}
I20260620 21:54:17.030007 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:17.029366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:36001)
0620 21:54:17.029827 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:17.029899 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":131,"threads_started":1}
I20260620 21:54:17.226795 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:17.226294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:55037)
0620 21:54:17.226626 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:17.226697 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":75,"threads_started":1}
I20260620 21:54:17.874459 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:17.874005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:50041)
0620 21:54:17.874298 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:17.874362 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
I20260620 21:54:18.031498 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:18.030990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:35165)
0620 21:54:18.031330 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:18.031393 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":80,"threads_started":1}
I20260620 21:54:18.228192 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:18.227731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:55867)
0620 21:54:18.228039 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:18.228099 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260620 21:54:18.875962 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:18.875461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:39381)
0620 21:54:18.875794 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:18.875861 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":83,"threads_started":1}
I20260620 21:54:19.032878 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:19.032447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:50799)
0620 21:54:19.032714 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:19.032780 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
I20260620 21:54:19.229460 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:19.228976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:56635)
0620 21:54:19.229269 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:19.229342 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":88,"threads_started":1}
I20260620 21:54:19.402673 17766 raft_consensus.cc:482] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:54:19.402799 17766 raft_consensus.cc:504] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabb641223ee4852bfb2b7e67242997a" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 33481 } } peers { permanent_uuid: "78b7ec3c91454fb08fec7e1824118efc" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 46345 } } peers { permanent_uuid: "53cd1d8a03dd401b913ec3b6d0fad6c8" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35883 } }
I20260620 21:54:19.403353 17766 leader_election.cc:288] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481), 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345)
W20260620 21:54:19.404073 17350 leader_election.cc:341] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 78b7ec3c91454fb08fec7e1824118efc (127.0.193.195:46345): Not found: Tablet not found: e53726a749944c0daa3ddb799c4b4c2f
W20260620 21:54:19.404208 17350 leader_election.cc:341] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabb641223ee4852bfb2b7e67242997a (127.0.193.194:33481): Not found: Tablet not found: e53726a749944c0daa3ddb799c4b4c2f
I20260620 21:54:19.404270 17350 leader_election.cc:302] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53cd1d8a03dd401b913ec3b6d0fad6c8; no voters: 78b7ec3c91454fb08fec7e1824118efc, dabb641223ee4852bfb2b7e67242997a
I20260620 21:54:19.404402 17766 raft_consensus.cc:2726] T e53726a749944c0daa3ddb799c4b4c2f P 53cd1d8a03dd401b913ec3b6d0fad6c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 21:54:19.877708 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:19.877081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:45921)
0620 21:54:19.877459 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:19.877598 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":83,"threads_started":1}
I20260620 21:54:20.034293 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:20.033776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:55401)
0620 21:54:20.034134 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:20.034198 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
I20260620 21:54:20.230908 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:20.230412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:46547)
0620 21:54:20.230729 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:20.230805 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":76,"threads_started":1}
I20260620 21:54:20.879428 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:20.878861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:43403)
0620 21:54:20.879270 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:20.879332 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
I20260620 21:54:21.035727 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:21.035305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:49929)
0620 21:54:21.035571 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:21.035631 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
I20260620 21:54:21.232270 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:21.231761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:44677)
0620 21:54:21.232109 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:21.232166 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":88,"threads_started":1}
I20260620 21:54:21.880712 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:21.880269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:38665)
0620 21:54:21.880550 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:21.880616 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
I20260620 21:54:22.036975 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:22.036556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:33529)
0620 21:54:22.036823 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:22.036879 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
I20260620 21:54:22.233493 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:22.233026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:44501)
0620 21:54:22.233343 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:22.233396 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":155,"threads_started":1}
I20260620 21:54:22.882385 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:22.881805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:40227)
0620 21:54:22.882205 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:22.882283 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":89,"threads_started":1}
I20260620 21:54:23.038273 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:23.037835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:60443)
0620 21:54:23.038112 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:23.038174 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
I20260620 21:54:23.235384 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:23.234780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:40773)
0620 21:54:23.235189 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:23.235285 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":90,"threads_started":1}
I20260620 21:54:23.883677 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:23.883175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:38697)
0620 21:54:23.883500 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:23.883575 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":84,"threads_started":1}
I20260620 21:54:24.039726 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:24.039269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:33719)
0620 21:54:24.039551 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:24.039628 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":74,"threads_started":1}
I20260620 21:54:24.236843 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:24.236347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:39905)
0620 21:54:24.236693 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:24.236747 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":91,"threads_started":1}
I20260620 21:54:24.885278 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:24.884756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.193:59871)
0620 21:54:24.885077 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:24.885174 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":85,"threads_started":1}
I20260620 21:54:25.041249 17783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:25.040810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.194:44773)
0620 21:54:25.041096 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:25.041156 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260620 21:54:25.226771 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 17292
I20260620 21:54:25.234061 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 17465
I20260620 21:54:25.238591 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:25.238091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36223 (local address 127.0.193.195:35201)
0620 21:54:25.238379 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:25.238444 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.254:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
I20260620 21:54:25.240767 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 17600
2026-06-20T21:54:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59451 ms)
[----------] 30 tests from SecurityITest (254397 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:25 dist-test-slave-kk5c krb5kdc[17787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:25 dist-test-slave-kk5c krb5kdc[17787](info): set up 2 sockets
Jun 20 21:54:25 dist-test-slave-kk5c krb5kdc[17787](info): commencing operation
krb5kdc: starting...
W20260620 21:54:26.335280 775 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:26 dist-test-slave-kk5c krb5kdc[17787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:54:26.468395 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36213
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:36213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:26.594187 17800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:26.594548 17800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:26.594595 17800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:26.599670 17800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:26.599767 17800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:26.599789 17800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:26.599808 17800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:26.599825 17800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:26.605966 17800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:36213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:26.607679 17800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:26.609272 17800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:26.621935 17806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:26.622040 17808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:26.622058 17805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:26.622285 17800 server_base.cc:1034] running on GCE node
I20260620 21:54:26.622983 17800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:26.624328 17800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:26.625537 17800 hybrid_clock.cc:648] HybridClock initialized: now 1781992466625522 us; error 94 us; skew 500 ppm
Jun 20 21:54:26 dist-test-slave-kk5c krb5kdc[17787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:26.628953 17800 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:54:26.630089 17800 webserver.cc:460] Webserver started at http://127.0.193.254:46243/ using document root <none> and password file <none>
I20260620 21:54:26.630716 17800 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:26.630815 17800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:26.631058 17800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:26.633543 17800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "c311778e0a5b4d06b1f7bada89a5ea3e"
format_stamp: "Formatted at 2026-06-20 21:54:26 on dist-test-slave-kk5c"
server_key: "f84dc85a3fab53b7bdc06898ca93858c"
server_key_iv: "59b0ba2066eb7171b499736595387af7"
server_key_version: "encryptionkey@0"
I20260620 21:54:26.634239 17800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "c311778e0a5b4d06b1f7bada89a5ea3e"
format_stamp: "Formatted at 2026-06-20 21:54:26 on dist-test-slave-kk5c"
server_key: "f84dc85a3fab53b7bdc06898ca93858c"
server_key_iv: "59b0ba2066eb7171b499736595387af7"
server_key_version: "encryptionkey@0"
I20260620 21:54:26.638478 17800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260620 21:54:26.641526 17815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:26.642969 17800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 21:54:26.643167 17800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "c311778e0a5b4d06b1f7bada89a5ea3e"
format_stamp: "Formatted at 2026-06-20 21:54:26 on dist-test-slave-kk5c"
server_key: "f84dc85a3fab53b7bdc06898ca93858c"
server_key_iv: "59b0ba2066eb7171b499736595387af7"
server_key_version: "encryptionkey@0"
I20260620 21:54:26.643443 17800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:26.659592 17800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:26.663368 17800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:26.663619 17800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:26.671104 17800 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:36213
I20260620 21:54:26.671133 17867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:36213 every 8 connection(s)
I20260620 21:54:26.672122 17800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:26.675027 17868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:26.676482 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 17800
I20260620 21:54:26.676635 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:26.676949 775 external_mini_cluster.cc:1428] Setting key d267e2701581799d97ea42b2e0b9afa6
I20260620 21:54:26.682126 17868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e: Bootstrap starting.
I20260620 21:54:26.684520 17868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:26.685300 17868 log.cc:824] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:54:26 dist-test-slave-kk5c krb5kdc[17787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:54:26.687932 17868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e: No bootstrap required, opened a new log
I20260620 21:54:26.692979 17868 raft_consensus.cc:348] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c311778e0a5b4d06b1f7bada89a5ea3e" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36213 } }
I20260620 21:54:26.693269 17868 raft_consensus.cc:374] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:26.693320 17868 raft_consensus.cc:729] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c311778e0a5b4d06b1f7bada89a5ea3e, State: Initialized, Role: FOLLOWER
I20260620 21:54:26.693353 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:26.678488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.0.193.254:36213)
0620 21:54:26.678963 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:26.678978 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:26.679014 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:54:26.679714 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:26.679732 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:26.680189 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:26.680600 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:26.681912 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:26.682948 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:26.683875 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:26.684352 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:26.684540 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:26.687588 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:26.687595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:26.687613 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:26.687645 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:26.689793 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:26.690342 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:26.690352 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:26.690359 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:26.690546 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:26.690900 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:26.690905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:26.690907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:26.691466 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:26.691689 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:26.691979 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 21:54:26.692322 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":145,"threads_started":1}
I20260620 21:54:26.693836 17868 consensus_queue.cc:260] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c311778e0a5b4d06b1f7bada89a5ea3e" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36213 } }
I20260620 21:54:26.694006 17868 raft_consensus.cc:388] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:26.694116 17868 raft_consensus.cc:482] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:26.694227 17868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:26.695276 17868 raft_consensus.cc:504] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c311778e0a5b4d06b1f7bada89a5ea3e" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36213 } }
I20260620 21:54:26.695657 17868 leader_election.cc:302] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c311778e0a5b4d06b1f7bada89a5ea3e; no voters:
I20260620 21:54:26.696024 17868 leader_election.cc:288] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:26.696185 17873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:26.696477 17873 raft_consensus.cc:686] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [term 1 LEADER]: Becoming Leader. State: Replica: c311778e0a5b4d06b1f7bada89a5ea3e, State: Running, Role: LEADER
I20260620 21:54:26.696890 17873 consensus_queue.cc:237] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c311778e0a5b4d06b1f7bada89a5ea3e" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36213 } }
I20260620 21:54:26.697310 17868 sys_catalog.cc:564] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:26.698774 17874 sys_catalog.cc:455] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c311778e0a5b4d06b1f7bada89a5ea3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c311778e0a5b4d06b1f7bada89a5ea3e" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36213 } } }
I20260620 21:54:26.698925 17874 sys_catalog.cc:458] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:26.699093 17875 sys_catalog.cc:455] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c311778e0a5b4d06b1f7bada89a5ea3e. Latest consensus state: current_term: 1 leader_uuid: "c311778e0a5b4d06b1f7bada89a5ea3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c311778e0a5b4d06b1f7bada89a5ea3e" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36213 } } }
I20260620 21:54:26.699229 17875 sys_catalog.cc:458] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:26.699707 17883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:26.702739 17883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:26.709038 17883 catalog_manager.cc:1347] Generated new cluster ID: bf08b39258164e8fa53a480cbd25c4c1
I20260620 21:54:26.709129 17883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:26.723097 17883 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:26.724537 17883 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:26.729125 17883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e: Generated new TSK 0
I20260620 21:54:26.729959 17883 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:54:26.763098 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36213
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:26.888603 17894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:26.888926 17894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:26.888972 17894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:26.895269 17894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:26.895370 17894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:26.895437 17894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:54:26.902190 17894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:26.903353 17894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:26.904362 17894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:54:26.914984 17894 server_base.cc:1034] running on GCE node
W20260620 21:54:26.914987 17900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:26.914987 17902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:26.915216 17899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:26.915746 17894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:26.916462 17894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:26.917714 17894 hybrid_clock.cc:648] HybridClock initialized: now 1781992466917694 us; error 36 us; skew 500 ppm
Jun 20 21:54:26 dist-test-slave-kk5c krb5kdc[17787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:26.921025 17894 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:54:26.922123 17894 webserver.cc:460] Webserver started at http://127.0.193.193:35239/ using document root <none> and password file <none>
I20260620 21:54:26.922695 17894 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:26.922765 17894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:26.922956 17894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:26.925209 17894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "86e630dc5c4844db8b860418f0392b00"
format_stamp: "Formatted at 2026-06-20 21:54:26 on dist-test-slave-kk5c"
server_key: "50b7557cab1e81c6d691939f23eece27"
server_key_iv: "7348adefcdefcc84e1abac0ceb3bd4ce"
server_key_version: "encryptionkey@0"
I20260620 21:54:26.925791 17894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "86e630dc5c4844db8b860418f0392b00"
format_stamp: "Formatted at 2026-06-20 21:54:26 on dist-test-slave-kk5c"
server_key: "50b7557cab1e81c6d691939f23eece27"
server_key_iv: "7348adefcdefcc84e1abac0ceb3bd4ce"
server_key_version: "encryptionkey@0"
I20260620 21:54:26.930076 17894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:54:26.932983 17909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:26.934557 17894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:54:26.934685 17894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "86e630dc5c4844db8b860418f0392b00"
format_stamp: "Formatted at 2026-06-20 21:54:26 on dist-test-slave-kk5c"
server_key: "50b7557cab1e81c6d691939f23eece27"
server_key_iv: "7348adefcdefcc84e1abac0ceb3bd4ce"
server_key_version: "encryptionkey@0"
I20260620 21:54:26.934865 17894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:26.955508 17894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:26.959159 17894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:26.959472 17894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:26.960157 17894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:26.961061 17894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:26.961110 17894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:26.961153 17894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:26.961167 17894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:26.972018 17894 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:37655
I20260620 21:54:26.972036 18022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:37655 every 8 connection(s)
I20260620 21:54:26.973070 17894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:54:26.982055 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 17894
I20260620 21:54:26.982209 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:54:26.982528 775 external_mini_cluster.cc:1428] Setting key 7a9d7f568134abecfcbbb9b509c4e40d
Jun 20 21:54:26 dist-test-slave-kk5c krb5kdc[17787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:54:26.989336 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:26.975147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:35203 (local address 127.0.193.254:36213)
0620 21:54:26.975370 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:26.975374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:26.976697 (+ 1323us) server_negotiation.cc:407] Connection header received
0620 21:54:26.977785 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:26.977788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:26.977855 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:26.977960 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:26.979250 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:26.979786 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:26.980958 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:26.981197 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:26.981309 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:26.984554 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:26.984557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:26.984559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:26.984584 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:26.986612 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:26.987359 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:26.987366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:26.987370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:26.987468 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:26.987873 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:26.987879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:26.987884 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:26.988175 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:26.988280 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:26.988934 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 21:54:26.989095 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260620 21:54:26.990147 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:26.975681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36213 (local address 127.0.193.193:35203)
0620 21:54:26.976484 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:26.976562 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:26.977542 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:26.978173 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:26.978180 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:26.978576 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:26.979069 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:26.979083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:26.979918 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:26.979927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:26.980818 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:26.980826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:26.981359 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:26.981363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:26.981538 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:26.982376 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:54:26.982396 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:26.984348 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:26.986774 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:26.986780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:26.986801 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:26.987159 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:26.987593 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:26.987595 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:26.987597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:26.987751 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:26.988285 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:26.988291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:26.988563 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 21:54:26.988812 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 21:54:26.989118 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":161,"threads_started":1}
I20260620 21:54:26.991562 18023 heartbeater.cc:347] Connected to a master server at 127.0.193.254:36213
I20260620 21:54:26.991842 18023 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:26.992372 18023 heartbeater.cc:511] Master 127.0.193.254:36213 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
I20260620 21:54:26.994068 17832 ts_manager.cc:194] Registered new tserver with Master: 86e630dc5c4844db8b860418f0392b00 (127.0.193.193:37655)
I20260620 21:54:26.995443 17832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:35203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:54:27.015100 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36213
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:27.130208 18028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:27.130514 18028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:27.130559 18028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:27.134856 18028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:27.134934 18028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:27.134995 18028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:54:27.140801 18028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:27.141937 18028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:27.142902 18028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:27.152881 18036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:27.152905 18033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:27.152988 18034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:27.153055 18028 server_base.cc:1034] running on GCE node
I20260620 21:54:27.153685 18028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:27.154418 18028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:27.155611 18028 hybrid_clock.cc:648] HybridClock initialized: now 1781992467155596 us; error 34 us; skew 500 ppm
Jun 20 21:54:27 dist-test-slave-kk5c krb5kdc[17787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:27.158805 18028 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:54:27.159950 18028 webserver.cc:460] Webserver started at http://127.0.193.194:37917/ using document root <none> and password file <none>
I20260620 21:54:27.160686 18028 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:27.160791 18028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:27.161051 18028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:27.163288 18028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "1881de142ed84a59baf0b84364d71146"
format_stamp: "Formatted at 2026-06-20 21:54:27 on dist-test-slave-kk5c"
server_key: "6dcc4232eb6ad731d74d71fb500b2dbd"
server_key_iv: "eefaf6392d107dabbc0bc49f0c65e53a"
server_key_version: "encryptionkey@0"
I20260620 21:54:27.163909 18028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "1881de142ed84a59baf0b84364d71146"
format_stamp: "Formatted at 2026-06-20 21:54:27 on dist-test-slave-kk5c"
server_key: "6dcc4232eb6ad731d74d71fb500b2dbd"
server_key_iv: "eefaf6392d107dabbc0bc49f0c65e53a"
server_key_version: "encryptionkey@0"
I20260620 21:54:27.168383 18028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:54:27.171051 18043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:27.172528 18028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:54:27.172684 18028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "1881de142ed84a59baf0b84364d71146"
format_stamp: "Formatted at 2026-06-20 21:54:27 on dist-test-slave-kk5c"
server_key: "6dcc4232eb6ad731d74d71fb500b2dbd"
server_key_iv: "eefaf6392d107dabbc0bc49f0c65e53a"
server_key_version: "encryptionkey@0"
I20260620 21:54:27.172816 18028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:27.205461 18028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:27.209090 18028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:27.209314 18028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:27.209957 18028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:27.210840 18028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:27.210889 18028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:27.210965 18028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:27.211004 18028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:27.220181 18028 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:34675
I20260620 21:54:27.220255 18156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:34675 every 8 connection(s)
I20260620 21:54:27.221122 18028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:54:27.222637 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 18028
I20260620 21:54:27.222761 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:54:27.223070 775 external_mini_cluster.cc:1428] Setting key 47e66818c140fd1bfd675bd17a210797
Jun 20 21:54:27 dist-test-slave-kk5c krb5kdc[17787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
I20260620 21:54:27.237227 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.223120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:52651 (local address 127.0.193.254:36213)
0620 21:54:27.223293 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:27.223299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:27.224989 (+ 1690us) server_negotiation.cc:407] Connection header received
0620 21:54:27.226008 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:27.226011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:27.226069 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:27.226148 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:27.227455 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.227993 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.229821 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.230060 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.230178 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.232984 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:27.232991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:27.232995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:27.233035 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:27.234790 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:27.235547 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:27.235551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:27.235555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:27.235636 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:27.236000 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:27.236003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:27.236005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:27.236248 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:27.236356 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:27.236908 (+ 552us) server_negotiation.cc:299] Negotiation successful
0620 21:54:27.237051 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:54:27.238073 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.223652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36213 (local address 127.0.193.194:52651)
0620 21:54:27.224759 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:27.224825 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:27.225767 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:27.226335 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:27.226343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:27.226711 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:27.227295 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:27.227313 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.228141 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.228144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:27.229666 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:27.229679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.230239 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.230243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:27.230414 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.231152 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:54:27.231170 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:27.232759 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:27.235011 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:27.235018 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:27.235039 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:27.235427 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:27.235733 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:27.235736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:27.235738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:27.235894 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:27.236361 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:27.236370 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:27.236644 (+ 274us) client_negotiation.cc:769] Sending connection context
0620 21:54:27.236845 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 21:54:27.237201 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":178,"threads_started":1}
I20260620 21:54:27.239437 18157 heartbeater.cc:347] Connected to a master server at 127.0.193.254:36213
I20260620 21:54:27.239720 18157 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:27.240489 18157 heartbeater.cc:511] Master 127.0.193.254:36213 requested a full tablet report, sending...
I20260620 21:54:27.241744 17832 ts_manager.cc:194] Registered new tserver with Master: 1881de142ed84a59baf0b84364d71146 (127.0.193.194:34675)
I20260620 21:54:27.242383 17832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:52651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:54:27.254882 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36213
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:27.369385 18162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:27.369705 18162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:27.369788 18162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:27.373999 18162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:27.374104 18162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:27.374198 18162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:54:27.380023 18162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:36213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:27.381192 18162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:27.382279 18162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:27.392161 18167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:27.392169 18170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:27.392169 18168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:27.392434 18162 server_base.cc:1034] running on GCE node
I20260620 21:54:27.392995 18162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:27.393718 18162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:27.394930 18162 hybrid_clock.cc:648] HybridClock initialized: now 1781992467394906 us; error 38 us; skew 500 ppm
Jun 20 21:54:27 dist-test-slave-kk5c krb5kdc[17787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:27.398015 18162 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:54:27.398995 18162 webserver.cc:460] Webserver started at http://127.0.193.195:41865/ using document root <none> and password file <none>
I20260620 21:54:27.399605 18162 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:27.399693 18162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:27.399894 18162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:27.402102 18162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "03974f56b26046f6b48748bec85c97ab"
format_stamp: "Formatted at 2026-06-20 21:54:27 on dist-test-slave-kk5c"
server_key: "97493b6f4bf93e4fa41d8e1906b67f96"
server_key_iv: "fb9d8ed957a6587660af89d98f1bdfc0"
server_key_version: "encryptionkey@0"
I20260620 21:54:27.402690 18162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "03974f56b26046f6b48748bec85c97ab"
format_stamp: "Formatted at 2026-06-20 21:54:27 on dist-test-slave-kk5c"
server_key: "97493b6f4bf93e4fa41d8e1906b67f96"
server_key_iv: "fb9d8ed957a6587660af89d98f1bdfc0"
server_key_version: "encryptionkey@0"
I20260620 21:54:27.406929 18162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 21:54:27.410066 18177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:27.411758 18162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:27.411916 18162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "03974f56b26046f6b48748bec85c97ab"
format_stamp: "Formatted at 2026-06-20 21:54:27 on dist-test-slave-kk5c"
server_key: "97493b6f4bf93e4fa41d8e1906b67f96"
server_key_iv: "fb9d8ed957a6587660af89d98f1bdfc0"
server_key_version: "encryptionkey@0"
I20260620 21:54:27.412060 18162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:27.423020 18162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:27.426537 18162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:27.426795 18162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:27.427503 18162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:27.428428 18162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:27.428500 18162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:27.428567 18162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:27.428614 18162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:27.438288 18162 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:42275
I20260620 21:54:27.438313 18290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:42275 every 8 connection(s)
I20260620 21:54:27.439471 18162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:54:27.442077 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 18162
I20260620 21:54:27.442183 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:54:27.442564 775 external_mini_cluster.cc:1428] Setting key bd63114561d314658e37a4332c9c55bc
Jun 20 21:54:27 dist-test-slave-kk5c krb5kdc[17787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:54:27.454548 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.441612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:56243 (local address 127.0.193.254:36213)
0620 21:54:27.441767 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:27.441772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:27.442919 (+ 1147us) server_negotiation.cc:407] Connection header received
0620 21:54:27.444037 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:27.444040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:27.444100 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:27.444178 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:27.445439 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.445978 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.447148 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.447444 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.447563 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.450434 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:27.450438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:27.450441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:27.450478 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:27.452212 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:27.452884 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:27.452888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:27.452889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:27.452972 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:27.453363 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:27.453367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:27.453368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:27.453573 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:27.453690 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:27.454240 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 21:54:27.454370 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:54:27.455605 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.442083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36213 (local address 127.0.193.195:56243)
0620 21:54:27.442695 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:27.442768 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:27.443813 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:27.444368 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:27.444377 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:27.444768 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:27.445284 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:27.445298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.446116 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.446120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:27.447014 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:27.447024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.447605 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.447608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:27.447787 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.448475 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:54:27.448492 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:27.450208 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:27.452388 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:27.452395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:27.452421 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:27.452703 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:27.453074 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:27.453080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:27.453083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:27.453241 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:27.453689 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:27.453698 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:27.453989 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 21:54:27.454196 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 21:54:27.454564 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":171,"threads_started":1}
I20260620 21:54:27.457074 18291 heartbeater.cc:347] Connected to a master server at 127.0.193.254:36213
I20260620 21:54:27.457345 18291 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:27.457926 18291 heartbeater.cc:511] Master 127.0.193.254:36213 requested a full tablet report, sending...
I20260620 21:54:27.459044 17832 ts_manager.cc:194] Registered new tserver with Master: 03974f56b26046f6b48748bec85c97ab (127.0.193.195:42275)
I20260620 21:54:27.459709 17832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:56243
I20260620 21:54:27.468767 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:54:27.479082 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.470753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.0.193.254:36213)
0620 21:54:27.470886 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:27.470889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:27.471060 (+ 171us) server_negotiation.cc:407] Connection header received
0620 21:54:27.471158 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:27.471161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:27.471258 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:27.471334 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:27.472119 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.472644 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.473555 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.473765 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.473865 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.475528 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:27.475531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:27.475533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:27.475557 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:27.477100 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:27.477721 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:27.477724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:27.477725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:27.477800 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:27.478201 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:27.478203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:27.478205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:27.478434 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:27.478543 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:27.478826 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 21:54:27.478925 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:54:27.482139 17832 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:54:27.485216 17832 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:54:27.500268 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.493762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:42275 (local address 127.0.0.1:33202)
0620 21:54:27.494102 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:27.494146 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:27.494276 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:27.495844 (+ 1568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:27.495846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:27.495858 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:27.496046 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:27.496058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.497545 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.497548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:27.498690 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:27.498702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.499704 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.499708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:27.499816 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.499852 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 21:54:27.499966 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 21:54:27.500092 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":52,"spinlock_wait_cycles":6272,"thread_start_us":117,"threads_started":1}
I20260620 21:54:27.500296 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.493847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:34675 (local address 127.0.0.1:34810)
0620 21:54:27.494103 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:27.494147 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:27.494276 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:27.495595 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:27.495601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:27.495622 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:27.495776 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:27.495783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.497159 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.497162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:27.498867 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:27.498877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.499976 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.499980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:27.500059 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.500079 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 21:54:27.500136 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 21:54:27.500193 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":91,"spinlock_wait_cycles":12928,"thread_start_us":118,"threads_started":1}
I20260620 21:54:27.500418 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.494633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34810 (local address 127.0.193.194:34675)
0620 21:54:27.495062 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:27.495072 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:27.495091 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:54:27.495173 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:27.495177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:27.495428 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:27.495927 (+ 499us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:27.496166 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.497008 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.499016 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.499851 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.499966 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.500103 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:27.500173 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 21:54:27.500252 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":108,"threads_started":1}
I20260620 21:54:27.500775 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.494499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:37655 (local address 127.0.0.1:44992)
0620 21:54:27.494789 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:27.494802 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:27.494884 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:27.495820 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:27.495826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:27.495845 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:27.495999 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:27.496007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.497443 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.497447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:27.498870 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:27.498895 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.500347 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.500351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:27.500436 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.500459 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:54:27.500539 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 21:54:27.500626 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":152,"spinlock_wait_cycles":768,"thread_start_us":97,"threads_started":1}
I20260620 21:54:27.501204 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.494865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.0.193.193:37655)
0620 21:54:27.495178 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:27.495205 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:27.495229 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:54:27.495305 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:27.495312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:27.495523 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:27.495667 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:27.496131 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.497322 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.499021 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.499851 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.499973 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.500103 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:27.500943 (+ 840us) server_negotiation.cc:299] Negotiation successful
0620 21:54:27.501067 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":144,"threads_started":1}
I20260620 21:54:27.502130 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.494603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.0.193.195:42275)
0620 21:54:27.495443 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:27.495453 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:27.495475 (+ 22us) server_negotiation.cc:407] Connection header received
0620 21:54:27.495542 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:27.495546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:27.495705 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:27.496042 (+ 337us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:27.496203 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.497368 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.498820 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.499579 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.499702 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.499861 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:27.501873 (+ 2012us) server_negotiation.cc:299] Negotiation successful
0620 21:54:27.501990 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":122,"threads_started":1}
I20260620 21:54:27.504020 18091 tablet_service.cc:1467] Processing CreateTablet for tablet b5d4ac69c1274eb7a47257f0f88317c7 (DEFAULT_TABLE table=test-table [id=8d35cde6466a41678965d86dca34964b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:27.504045 17957 tablet_service.cc:1467] Processing CreateTablet for tablet b5d4ac69c1274eb7a47257f0f88317c7 (DEFAULT_TABLE table=test-table [id=8d35cde6466a41678965d86dca34964b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:27.505050 18091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d4ac69c1274eb7a47257f0f88317c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:27.505079 17957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d4ac69c1274eb7a47257f0f88317c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:27.504794 18225 tablet_service.cc:1467] Processing CreateTablet for tablet b5d4ac69c1274eb7a47257f0f88317c7 (DEFAULT_TABLE table=test-table [id=8d35cde6466a41678965d86dca34964b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:27.505709 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d4ac69c1274eb7a47257f0f88317c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:27.512598 18308 tablet_bootstrap.cc:492] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146: Bootstrap starting.
I20260620 21:54:27.512876 18309 tablet_bootstrap.cc:492] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00: Bootstrap starting.
I20260620 21:54:27.513302 18310 tablet_bootstrap.cc:492] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab: Bootstrap starting.
I20260620 21:54:27.515484 18309 tablet_bootstrap.cc:654] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:27.515535 18308 tablet_bootstrap.cc:654] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:27.516402 18309 log.cc:824] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:27.516403 18308 log.cc:824] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:27.517254 18310 tablet_bootstrap.cc:654] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:27.518525 18310 log.cc:824] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:27.518677 18309 tablet_bootstrap.cc:492] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00: No bootstrap required, opened a new log
I20260620 21:54:27.518903 18309 ts_tablet_manager.cc:1397] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:54:27.520414 18310 tablet_bootstrap.cc:492] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab: No bootstrap required, opened a new log
I20260620 21:54:27.520607 18310 ts_tablet_manager.cc:1397] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 21:54:27.522583 18309 raft_consensus.cc:348] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03974f56b26046f6b48748bec85c97ab" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 42275 } } peers { permanent_uuid: "1881de142ed84a59baf0b84364d71146" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34675 } } peers { permanent_uuid: "86e630dc5c4844db8b860418f0392b00" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37655 } }
I20260620 21:54:27.522837 18309 raft_consensus.cc:374] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:27.522933 18309 raft_consensus.cc:729] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e630dc5c4844db8b860418f0392b00, State: Initialized, Role: FOLLOWER
I20260620 21:54:27.523452 18309 consensus_queue.cc:260] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03974f56b26046f6b48748bec85c97ab" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 42275 } } peers { permanent_uuid: "1881de142ed84a59baf0b84364d71146" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34675 } } peers { permanent_uuid: "86e630dc5c4844db8b860418f0392b00" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37655 } }
I20260620 21:54:27.523559 18310 raft_consensus.cc:348] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03974f56b26046f6b48748bec85c97ab" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 42275 } } peers { permanent_uuid: "1881de142ed84a59baf0b84364d71146" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34675 } } peers { permanent_uuid: "86e630dc5c4844db8b860418f0392b00" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37655 } }
I20260620 21:54:27.523852 18310 raft_consensus.cc:374] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:27.523913 18310 raft_consensus.cc:729] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03974f56b26046f6b48748bec85c97ab, State: Initialized, Role: FOLLOWER
I20260620 21:54:27.524326 18023 heartbeater.cc:503] Master 127.0.193.254:36213 was elected leader, sending a full tablet report...
I20260620 21:54:27.524370 18310 consensus_queue.cc:260] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03974f56b26046f6b48748bec85c97ab" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 42275 } } peers { permanent_uuid: "1881de142ed84a59baf0b84364d71146" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34675 } } peers { permanent_uuid: "86e630dc5c4844db8b860418f0392b00" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37655 } }
I20260620 21:54:27.524471 18308 tablet_bootstrap.cc:492] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146: No bootstrap required, opened a new log
I20260620 21:54:27.524539 18309 ts_tablet_manager.cc:1428] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:54:27.524668 18308 ts_tablet_manager.cc:1397] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260620 21:54:27.525282 18310 ts_tablet_manager.cc:1428] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 21:54:27.525619 18291 heartbeater.cc:503] Master 127.0.193.254:36213 was elected leader, sending a full tablet report...
I20260620 21:54:27.528617 18308 raft_consensus.cc:348] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03974f56b26046f6b48748bec85c97ab" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 42275 } } peers { permanent_uuid: "1881de142ed84a59baf0b84364d71146" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34675 } } peers { permanent_uuid: "86e630dc5c4844db8b860418f0392b00" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37655 } }
I20260620 21:54:27.528970 18308 raft_consensus.cc:374] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:27.529038 18308 raft_consensus.cc:729] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1881de142ed84a59baf0b84364d71146, State: Initialized, Role: FOLLOWER
I20260620 21:54:27.529495 18308 consensus_queue.cc:260] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03974f56b26046f6b48748bec85c97ab" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 42275 } } peers { permanent_uuid: "1881de142ed84a59baf0b84364d71146" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34675 } } peers { permanent_uuid: "86e630dc5c4844db8b860418f0392b00" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37655 } }
I20260620 21:54:27.530169 18157 heartbeater.cc:503] Master 127.0.193.254:36213 was elected leader, sending a full tablet report...
I20260620 21:54:27.530447 18308 ts_tablet_manager.cc:1428] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260620 21:54:27.690734 18292 tablet.cc:2369] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:54:27.722396 18158 tablet.cc:2369] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:54:27.724685 18024 tablet.cc:2369] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:54:27.858824 18314 raft_consensus.cc:482] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:54:27.859143 18314 raft_consensus.cc:504] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03974f56b26046f6b48748bec85c97ab" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 42275 } } peers { permanent_uuid: "1881de142ed84a59baf0b84364d71146" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34675 } } peers { permanent_uuid: "86e630dc5c4844db8b860418f0392b00" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37655 } }
I20260620 21:54:27.860443 18314 leader_election.cc:288] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03974f56b26046f6b48748bec85c97ab (127.0.193.195:42275), 1881de142ed84a59baf0b84364d71146 (127.0.193.194:34675)
I20260620 21:54:27.866598 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.860941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:50291 (local address 127.0.193.194:34675)
0620 21:54:27.861097 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:27.861104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:27.861500 (+ 396us) server_negotiation.cc:407] Connection header received
0620 21:54:27.861640 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:27.861645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:27.861701 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:27.861768 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:27.862059 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.862797 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.865055 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.865656 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.865800 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.865855 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:27.866364 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 21:54:27.866459 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 21:54:27.866608 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.860913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:36179 (local address 127.0.193.195:42275)
0620 21:54:27.861130 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:27.861137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:27.861194 (+ 57us) server_negotiation.cc:407] Connection header received
0620 21:54:27.861376 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:27.861382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:27.861436 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:27.861542 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:27.861897 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.862712 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.865017 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.865686 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.865800 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.865855 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:27.866364 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 21:54:27.866484 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 21:54:27.866621 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.860754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:42275 (local address 127.0.193.193:36179)
0620 21:54:27.861097 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:27.861114 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:27.861239 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:27.861580 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:27.861586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:27.861605 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:27.861762 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:27.861772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.862841 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.862868 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:27.864690 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:27.864700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.865853 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.865857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:27.866135 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.866186 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 21:54:27.866281 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 21:54:27.866406 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":246,"thread_start_us":128,"threads_started":1}
I20260620 21:54:27.866621 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.860754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:34675 (local address 127.0.193.193:50291)
0620 21:54:27.861385 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:27.861397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:27.861499 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:27.861808 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:27.861813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:27.861836 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:27.861944 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:27.861949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.862906 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.862909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:27.864685 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:27.864700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.865853 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.865857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:27.866139 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.866191 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 21:54:27.866281 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 21:54:27.866406 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":697,"thread_start_us":127,"threads_started":1}
I20260620 21:54:27.867594 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d4ac69c1274eb7a47257f0f88317c7" candidate_uuid: "86e630dc5c4844db8b860418f0392b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1881de142ed84a59baf0b84364d71146" is_pre_election: true
I20260620 21:54:27.867568 18245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d4ac69c1274eb7a47257f0f88317c7" candidate_uuid: "86e630dc5c4844db8b860418f0392b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03974f56b26046f6b48748bec85c97ab" is_pre_election: true
I20260620 21:54:27.867931 18111 raft_consensus.cc:2454] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86e630dc5c4844db8b860418f0392b00 in term 0.
I20260620 21:54:27.867931 18245 raft_consensus.cc:2454] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86e630dc5c4844db8b860418f0392b00 in term 0.
I20260620 21:54:27.868522 17911 leader_election.cc:302] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03974f56b26046f6b48748bec85c97ab, 86e630dc5c4844db8b860418f0392b00; no voters:
I20260620 21:54:27.868788 18314 raft_consensus.cc:2781] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:54:27.868916 18314 raft_consensus.cc:482] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:54:27.868986 18314 raft_consensus.cc:3037] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:27.870249 18314 raft_consensus.cc:504] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03974f56b26046f6b48748bec85c97ab" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 42275 } } peers { permanent_uuid: "1881de142ed84a59baf0b84364d71146" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34675 } } peers { permanent_uuid: "86e630dc5c4844db8b860418f0392b00" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37655 } }
I20260620 21:54:27.870661 18314 leader_election.cc:288] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [CANDIDATE]: Term 1 election: Requested vote from peers 03974f56b26046f6b48748bec85c97ab (127.0.193.195:42275), 1881de142ed84a59baf0b84364d71146 (127.0.193.194:34675)
I20260620 21:54:27.871043 18245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d4ac69c1274eb7a47257f0f88317c7" candidate_uuid: "86e630dc5c4844db8b860418f0392b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03974f56b26046f6b48748bec85c97ab"
I20260620 21:54:27.871083 18111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d4ac69c1274eb7a47257f0f88317c7" candidate_uuid: "86e630dc5c4844db8b860418f0392b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1881de142ed84a59baf0b84364d71146"
I20260620 21:54:27.871220 18245 raft_consensus.cc:3037] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:27.871246 18111 raft_consensus.cc:3037] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:27.872295 18111 raft_consensus.cc:2454] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86e630dc5c4844db8b860418f0392b00 in term 1.
I20260620 21:54:27.872318 18245 raft_consensus.cc:2454] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86e630dc5c4844db8b860418f0392b00 in term 1.
I20260620 21:54:27.872640 17913 leader_election.cc:302] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1881de142ed84a59baf0b84364d71146, 86e630dc5c4844db8b860418f0392b00; no voters:
I20260620 21:54:27.872836 18314 raft_consensus.cc:2781] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:27.873126 18314 raft_consensus.cc:686] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [term 1 LEADER]: Becoming Leader. State: Replica: 86e630dc5c4844db8b860418f0392b00, State: Running, Role: LEADER
I20260620 21:54:27.873510 18314 consensus_queue.cc:237] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03974f56b26046f6b48748bec85c97ab" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 42275 } } peers { permanent_uuid: "1881de142ed84a59baf0b84364d71146" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34675 } } peers { permanent_uuid: "86e630dc5c4844db8b860418f0392b00" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37655 } }
I20260620 21:54:27.877141 17832 catalog_manager.cc:5495] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86e630dc5c4844db8b860418f0392b00 (127.0.193.193). New cstate: current_term: 1 leader_uuid: "86e630dc5c4844db8b860418f0392b00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03974f56b26046f6b48748bec85c97ab" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 42275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1881de142ed84a59baf0b84364d71146" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86e630dc5c4844db8b860418f0392b00" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37655 } health_report { overall_health: HEALTHY } } }
I20260620 21:54:27.957032 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:27.952883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.0.193.193:37655)
0620 21:54:27.953109 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:27.953113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:27.953126 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:54:27.953203 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:27.953205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:27.953260 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:27.953385 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:54:27.953665 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.954237 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.955233 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:27.955482 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:27.955618 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:27.955989 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:54:27.956472 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:54:27.956760 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:27.956802 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 21:54:27.956874 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260620 21:54:27.963497 18111 raft_consensus.cc:1261] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146 [term 1 FOLLOWER]: Refusing update from remote peer 86e630dc5c4844db8b860418f0392b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:54:27.963541 18245 raft_consensus.cc:1261] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab [term 1 FOLLOWER]: Refusing update from remote peer 86e630dc5c4844db8b860418f0392b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:54:27.964322 18319 consensus_queue.cc:1035] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1881de142ed84a59baf0b84364d71146" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 34675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:54:27.964537 18314 consensus_queue.cc:1035] T b5d4ac69c1274eb7a47257f0f88317c7 P 86e630dc5c4844db8b860418f0392b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03974f56b26046f6b48748bec85c97ab" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 42275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:54:27.975869 18322 mvcc.cc:205] Tried to move back new op lower bound from 7299041148770570240 to 7299041148412977152. Current Snapshot: MvccSnapshot[applied={T|T < 7299041148770570240}]
I20260620 21:54:27.977651 18323 mvcc.cc:205] Tried to move back new op lower bound from 7299041148770570240 to 7299041148412977152. Current Snapshot: MvccSnapshot[applied={T|T < 7299041148770570240}]
I20260620 21:54:27.987879 17830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:54:27.988071 17830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:54:27.990602 17830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c311778e0a5b4d06b1f7bada89a5ea3e: Sending DeleteTablet for 3 replicas of tablet b5d4ac69c1274eb7a47257f0f88317c7
I20260620 21:54:27.991528 17957 tablet_service.cc:1514] Processing DeleteTablet for tablet b5d4ac69c1274eb7a47257f0f88317c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:27 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:44992
I20260620 21:54:27.991535 18225 tablet_service.cc:1514] Processing DeleteTablet for tablet b5d4ac69c1274eb7a47257f0f88317c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:27 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:33202
I20260620 21:54:27.992069 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 17894
I20260620 21:54:27.992409 18091 tablet_service.cc:1514] Processing DeleteTablet for tablet b5d4ac69c1274eb7a47257f0f88317c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:27 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:34810
I20260620 21:54:27.992964 18333 tablet_replica.cc:330] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146: stopping tablet replica
I20260620 21:54:27.993315 18333 raft_consensus.cc:2229] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:54:27.993675 18333 raft_consensus.cc:2258] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:54:27.994125 18331 tablet_replica.cc:330] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab: stopping tablet replica
I20260620 21:54:27.994428 18331 raft_consensus.cc:2229] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:54:27.994701 18331 raft_consensus.cc:2258] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:54:27.995462 18333 ts_tablet_manager.cc:1905] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:54:27.996042 18331 ts_tablet_manager.cc:1905] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:54:27.998553 18331 ts_tablet_manager.cc:1918] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:54:27.998687 18331 log.cc:1196] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/b5d4ac69c1274eb7a47257f0f88317c7
I20260620 21:54:27.998992 18331 ts_tablet_manager.cc:1939] T b5d4ac69c1274eb7a47257f0f88317c7 P 03974f56b26046f6b48748bec85c97ab: Deleting consensus metadata
I20260620 21:54:27.999359 18333 ts_tablet_manager.cc:1918] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:54:27.999470 18333 log.cc:1196] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/b5d4ac69c1274eb7a47257f0f88317c7
I20260620 21:54:27.999802 18333 ts_tablet_manager.cc:1939] T b5d4ac69c1274eb7a47257f0f88317c7 P 1881de142ed84a59baf0b84364d71146: Deleting consensus metadata
I20260620 21:54:28.000639 17817 catalog_manager.cc:4841] TS 03974f56b26046f6b48748bec85c97ab (127.0.193.195:42275): tablet b5d4ac69c1274eb7a47257f0f88317c7 (table test-table [id=8d35cde6466a41678965d86dca34964b]) successfully deleted
W20260620 21:54:28.001297 17819 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:37655 (error 108)
I20260620 21:54:28.001528 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 18028
I20260620 21:54:28.001983 17819 catalog_manager.cc:4841] TS 1881de142ed84a59baf0b84364d71146 (127.0.193.194:34675): tablet b5d4ac69c1274eb7a47257f0f88317c7 (table test-table [id=8d35cde6466a41678965d86dca34964b]) successfully deleted
I20260620 21:54:28.003118 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:28.002759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:37655 (local address 127.0.0.1:45000)
0620 21:54:28.002962 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:28.003034 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
W20260620 21:54:28.003306 17819 catalog_manager.cc:4584] TS 86e630dc5c4844db8b860418f0392b00 (127.0.193.193:37655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5d4ac69c1274eb7a47257f0f88317c7: Network error: Client connection negotiation failed: client connection to 127.0.193.193:37655: connect: Connection refused (error 111)
I20260620 21:54:28.008270 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 18162
I20260620 21:54:28.017061 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 17800
2026-06-20T21:54:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2781 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T21:54:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:28Z Disabled control of system clock
I20260620 21:54:28.058123 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:39821
--webserver_interface=127.0.193.254
--webserver_port=0
--builtin_ntp_servers=127.0.193.212:36225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:39821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:54:28.177848 18339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:28.178131 18339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:28.178169 18339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:28.182300 18339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:28.182385 18339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:28.182402 18339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:28.182420 18339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:28.182436 18339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:28.187582 18339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:39821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:39821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:28.188650 18339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:28.189985 18339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:28.198599 18344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:28.198608 18347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:28.198683 18345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:28.199083 18339 server_base.cc:1034] running on GCE node
I20260620 21:54:28.199700 18339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:28.200794 18339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:28.202021 18339 hybrid_clock.cc:648] HybridClock initialized: now 1781992468201991 us; error 45 us; skew 500 ppm
I20260620 21:54:28.204175 18339 webserver.cc:460] Webserver started at http://127.0.193.254:43659/ using document root <none> and password file <none>
I20260620 21:54:28.204777 18339 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:28.204882 18339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:28.205123 18339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:28.207324 18339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "7450e7ed7f444d418e44e848b7de2a00"
format_stamp: "Formatted at 2026-06-20 21:54:28 on dist-test-slave-kk5c"
server_key: "4fb25ee35f0a21a922b26a476930e021"
server_key_iv: "167258917c6e16279f14b511268e000b"
server_key_version: "encryptionkey@0"
I20260620 21:54:28.207931 18339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "7450e7ed7f444d418e44e848b7de2a00"
format_stamp: "Formatted at 2026-06-20 21:54:28 on dist-test-slave-kk5c"
server_key: "4fb25ee35f0a21a922b26a476930e021"
server_key_iv: "167258917c6e16279f14b511268e000b"
server_key_version: "encryptionkey@0"
I20260620 21:54:28.211898 18339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:54:28.214509 18353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:28.216023 18339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:28.216184 18339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "7450e7ed7f444d418e44e848b7de2a00"
format_stamp: "Formatted at 2026-06-20 21:54:28 on dist-test-slave-kk5c"
server_key: "4fb25ee35f0a21a922b26a476930e021"
server_key_iv: "167258917c6e16279f14b511268e000b"
server_key_version: "encryptionkey@0"
I20260620 21:54:28.216311 18339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:28.226824 18339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:28.227644 18339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:28.227856 18339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:28.235554 18339 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:39821
I20260620 21:54:28.235601 18405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:39821 every 8 connection(s)
I20260620 21:54:28.236563 18339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:28.239902 18406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:28.245240 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 18339
I20260620 21:54:28.245426 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:28.245719 775 external_mini_cluster.cc:1428] Setting key 659874c975200b830898406d431aca0b
I20260620 21:54:28.245937 18406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00: Bootstrap starting.
I20260620 21:54:28.248853 18406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:28.249840 18406 log.cc:824] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:28.252426 18406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00: No bootstrap required, opened a new log
I20260620 21:54:28.256963 18406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7450e7ed7f444d418e44e848b7de2a00" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39821 } }
I20260620 21:54:28.257249 18406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:28.257300 18406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7450e7ed7f444d418e44e848b7de2a00, State: Initialized, Role: FOLLOWER
I20260620 21:54:28.257553 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:28.247395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.0.193.254:39821)
0620 21:54:28.248147 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:28.248168 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:28.248212 (+ 44us) server_negotiation.cc:407] Connection header received
0620 21:54:28.249153 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:28.249184 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:28.249730 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:28.250132 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:28.250679 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.252217 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.253295 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.254015 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.254189 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:28.255113 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:28.255122 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:28.255140 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:28.255748 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:28.255833 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:28.255870 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:28.255997 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:28.256166 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 21:54:28.256464 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":147,"threads_started":1}
I20260620 21:54:28.258040 18406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7450e7ed7f444d418e44e848b7de2a00" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39821 } }
I20260620 21:54:28.258230 18406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:28.258298 18406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:28.258386 18406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:28.259522 18406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7450e7ed7f444d418e44e848b7de2a00" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39821 } }
I20260620 21:54:28.259917 18406 leader_election.cc:302] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7450e7ed7f444d418e44e848b7de2a00; no voters:
I20260620 21:54:28.260286 18406 leader_election.cc:288] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:28.260385 18411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:28.260675 18411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [term 1 LEADER]: Becoming Leader. State: Replica: 7450e7ed7f444d418e44e848b7de2a00, State: Running, Role: LEADER
I20260620 21:54:28.261000 18411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7450e7ed7f444d418e44e848b7de2a00" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39821 } }
I20260620 21:54:28.261545 18406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:28.262655 18413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7450e7ed7f444d418e44e848b7de2a00. Latest consensus state: current_term: 1 leader_uuid: "7450e7ed7f444d418e44e848b7de2a00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7450e7ed7f444d418e44e848b7de2a00" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39821 } } }
I20260620 21:54:28.262791 18413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:28.263098 18412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7450e7ed7f444d418e44e848b7de2a00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7450e7ed7f444d418e44e848b7de2a00" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 39821 } } }
I20260620 21:54:28.263218 18412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:28.263340 18420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:28.266296 18420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:28.271929 18420 catalog_manager.cc:1347] Generated new cluster ID: 232aad9250c24268b478d63fc8a4ea71
I20260620 21:54:28.272027 18420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:28.307375 18420 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:28.308928 18420 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:28.323004 18420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00: Generated new TSK 0
I20260620 21:54:28.323975 18420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:28.333215 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:39821
--builtin_ntp_servers=127.0.193.212:36225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:54:28.456749 18430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:28.457094 18430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:28.457178 18430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:28.461979 18430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:28.462082 18430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:28.462189 18430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:54:28.467490 18430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:39821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:28.468721 18430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:28.470232 18430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:28.480086 18438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:28.480086 18435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:28.480120 18436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:28.480377 18430 server_base.cc:1034] running on GCE node
I20260620 21:54:28.480885 18430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:28.481559 18430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:28.482722 18430 hybrid_clock.cc:648] HybridClock initialized: now 1781992468482705 us; error 32 us; skew 500 ppm
I20260620 21:54:28.485044 18430 webserver.cc:460] Webserver started at http://127.0.193.193:42413/ using document root <none> and password file <none>
I20260620 21:54:28.485716 18430 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:28.485790 18430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:28.486038 18430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:28.488480 18430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "be23c1ecfb3943debf80d9afecad8feb"
format_stamp: "Formatted at 2026-06-20 21:54:28 on dist-test-slave-kk5c"
server_key: "a83bc1d4750f32f85b21cce350e3f905"
server_key_iv: "d188ff156b52542f241a3b18400b404b"
server_key_version: "encryptionkey@0"
I20260620 21:54:28.489168 18430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "be23c1ecfb3943debf80d9afecad8feb"
format_stamp: "Formatted at 2026-06-20 21:54:28 on dist-test-slave-kk5c"
server_key: "a83bc1d4750f32f85b21cce350e3f905"
server_key_iv: "d188ff156b52542f241a3b18400b404b"
server_key_version: "encryptionkey@0"
I20260620 21:54:28.493422 18430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:54:28.496284 18444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:28.497781 18430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 21:54:28.497943 18430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "be23c1ecfb3943debf80d9afecad8feb"
format_stamp: "Formatted at 2026-06-20 21:54:28 on dist-test-slave-kk5c"
server_key: "a83bc1d4750f32f85b21cce350e3f905"
server_key_iv: "d188ff156b52542f241a3b18400b404b"
server_key_version: "encryptionkey@0"
I20260620 21:54:28.498080 18430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:28.509460 18430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:28.510192 18430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:28.510403 18430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:28.511013 18430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:28.512063 18430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:28.512135 18430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:28.512207 18430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:28.512249 18430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:28.521450 18430 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:39519
I20260620 21:54:28.521476 18557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:39519 every 8 connection(s)
I20260620 21:54:28.522429 18430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:54:28.530756 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 18430
I20260620 21:54:28.530913 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:54:28.531275 775 external_mini_cluster.cc:1428] Setting key 8211ebfe5f2518d2710be6c97ac9d32f
I20260620 21:54:28.533242 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:28.524553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:32889 (local address 127.0.193.254:39821)
0620 21:54:28.524725 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:28.524729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:28.526058 (+ 1329us) server_negotiation.cc:407] Connection header received
0620 21:54:28.527557 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:28.527560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:28.527625 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:28.527735 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:28.528700 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.529240 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.530421 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.530652 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.530768 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:28.532524 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:28.532528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:28.532531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:28.532549 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:28.532578 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:28.532588 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:28.532683 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:28.533013 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 21:54:28.533124 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 21:54:28.534248 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:39821
--builtin_ntp_servers=127.0.193.212:36225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:54:28.534693 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:28.525014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39821 (local address 127.0.193.193:32889)
0620 21:54:28.525832 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:28.525875 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:28.527304 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:28.527922 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:28.527930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:28.528024 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:28.528540 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:28.528553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.529376 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.529387 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:28.530289 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:28.530298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.530818 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.530822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:28.531062 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:28.532054 (+ 992us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:54:28.532062 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:28.532109 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:54:28.532234 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:28.532764 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:28.532775 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:28.532782 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:54:28.532964 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 21:54:28.533383 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":148,"threads_started":1}
I20260620 21:54:28.536206 18558 heartbeater.cc:347] Connected to a master server at 127.0.193.254:39821
I20260620 21:54:28.536495 18558 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:28.537140 18558 heartbeater.cc:511] Master 127.0.193.254:39821 requested a full tablet report, sending...
I20260620 21:54:28.539079 18370 ts_manager.cc:194] Registered new tserver with Master: be23c1ecfb3943debf80d9afecad8feb (127.0.193.193:39519)
I20260620 21:54:28.540354 18370 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.193.193:32889
W20260620 21:54:28.658974 18561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:28.659312 18561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:28.659359 18561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:28.664150 18561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:28.664227 18561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:28.664307 18561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:54:28.669474 18561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:39821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:28.670643 18561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:28.672072 18561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:28.682854 18566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:28.682845 18569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:28.682845 18567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:28.683470 18561 server_base.cc:1034] running on GCE node
I20260620 21:54:28.684010 18561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:28.684736 18561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:28.685974 18561 hybrid_clock.cc:648] HybridClock initialized: now 1781992468685954 us; error 34 us; skew 500 ppm
I20260620 21:54:28.688285 18561 webserver.cc:460] Webserver started at http://127.0.193.194:42553/ using document root <none> and password file <none>
I20260620 21:54:28.688925 18561 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:28.688994 18561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:28.689234 18561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:28.691574 18561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "52bf9c42dd864f89b122abc7c7403675"
format_stamp: "Formatted at 2026-06-20 21:54:28 on dist-test-slave-kk5c"
server_key: "80c1f29f3ea3da315aa593eba855acbf"
server_key_iv: "791a774e45392e7bf5f4bfc2e5fbdd62"
server_key_version: "encryptionkey@0"
I20260620 21:54:28.692201 18561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "52bf9c42dd864f89b122abc7c7403675"
format_stamp: "Formatted at 2026-06-20 21:54:28 on dist-test-slave-kk5c"
server_key: "80c1f29f3ea3da315aa593eba855acbf"
server_key_iv: "791a774e45392e7bf5f4bfc2e5fbdd62"
server_key_version: "encryptionkey@0"
I20260620 21:54:28.696331 18561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:54:28.699162 18575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:28.700810 18561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:54:28.701049 18561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "52bf9c42dd864f89b122abc7c7403675"
format_stamp: "Formatted at 2026-06-20 21:54:28 on dist-test-slave-kk5c"
server_key: "80c1f29f3ea3da315aa593eba855acbf"
server_key_iv: "791a774e45392e7bf5f4bfc2e5fbdd62"
server_key_version: "encryptionkey@0"
I20260620 21:54:28.701185 18561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:28.738981 18561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:28.739948 18561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:28.740180 18561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:28.740852 18561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:28.741895 18561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:28.741959 18561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:28.742002 18561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:28.742044 18561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:28.753481 18561 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:39029
I20260620 21:54:28.753540 18688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:39029 every 8 connection(s)
I20260620 21:54:28.754647 18561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:54:28.764137 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 18561
I20260620 21:54:28.764240 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:54:28.764649 775 external_mini_cluster.cc:1428] Setting key aaebd8b51489f01b708fb9c1827f8695
I20260620 21:54:28.766569 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:28.757570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:47071 (local address 127.0.193.254:39821)
0620 21:54:28.757747 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:28.757751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:28.759001 (+ 1250us) server_negotiation.cc:407] Connection header received
0620 21:54:28.760579 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:28.760582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:28.760649 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:28.760753 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:28.761730 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.762401 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.763842 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.764301 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.764434 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:28.766045 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:28.766048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:28.766050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:28.766074 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:28.766095 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:28.766103 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:28.766190 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:28.766360 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 21:54:28.766428 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 21:54:28.766880 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:39821
--builtin_ntp_servers=127.0.193.212:36225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:54:28.767850 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:28.757931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39821 (local address 127.0.193.194:47071)
0620 21:54:28.758791 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:28.758848 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:28.760337 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:28.760951 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:28.760958 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:28.761048 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:28.761546 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:28.761558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.762575 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.762578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:28.763713 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:28.763724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.764498 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.764503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:28.764694 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:28.765643 (+ 949us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:54:28.765645 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:28.765692 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:54:28.765767 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:28.766194 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:28.766204 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:28.766211 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:54:28.766350 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 21:54:28.766683 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":135,"threads_started":1}
I20260620 21:54:28.769151 18689 heartbeater.cc:347] Connected to a master server at 127.0.193.254:39821
I20260620 21:54:28.769423 18689 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:28.770149 18689 heartbeater.cc:511] Master 127.0.193.254:39821 requested a full tablet report, sending...
I20260620 21:54:28.771919 18370 ts_manager.cc:194] Registered new tserver with Master: 52bf9c42dd864f89b122abc7c7403675 (127.0.193.194:39029)
I20260620 21:54:28.772573 18370 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.193.194:47071
W20260620 21:54:28.890887 18692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:28.891278 18692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:28.891367 18692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:28.896188 18692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:28.896327 18692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:28.896436 18692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:54:28.901674 18692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:36225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:39821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:28.902966 18692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:28.904464 18692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:28.914644 18700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:28.914914 18692 server_base.cc:1034] running on GCE node
W20260620 21:54:28.914647 18698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:28.914644 18697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:28.915614 18692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:28.916244 18692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:28.917661 18692 hybrid_clock.cc:648] HybridClock initialized: now 1781992468917627 us; error 42 us; skew 500 ppm
I20260620 21:54:28.919872 18692 webserver.cc:460] Webserver started at http://127.0.193.195:36291/ using document root <none> and password file <none>
I20260620 21:54:28.920531 18692 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:28.920626 18692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:28.920864 18692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:28.923053 18692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2"
format_stamp: "Formatted at 2026-06-20 21:54:28 on dist-test-slave-kk5c"
server_key: "52ce693d670474999f726ccd1e42ab13"
server_key_iv: "ff00eef8f222552133bd6853e07a392a"
server_key_version: "encryptionkey@0"
I20260620 21:54:28.923769 18692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2"
format_stamp: "Formatted at 2026-06-20 21:54:28 on dist-test-slave-kk5c"
server_key: "52ce693d670474999f726ccd1e42ab13"
server_key_iv: "ff00eef8f222552133bd6853e07a392a"
server_key_version: "encryptionkey@0"
I20260620 21:54:28.928153 18692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:54:28.931344 18706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:28.932871 18692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:28.933056 18692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2"
format_stamp: "Formatted at 2026-06-20 21:54:28 on dist-test-slave-kk5c"
server_key: "52ce693d670474999f726ccd1e42ab13"
server_key_iv: "ff00eef8f222552133bd6853e07a392a"
server_key_version: "encryptionkey@0"
I20260620 21:54:28.933204 18692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:28.949059 18692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:28.949891 18692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:28.950126 18692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:28.950817 18692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:28.951989 18692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:28.952071 18692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:28.952152 18692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:28.952201 18692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:28.962251 18692 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:34131
I20260620 21:54:28.962378 18819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:34131 every 8 connection(s)
I20260620 21:54:28.963464 18692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:54:28.964500 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 18692
I20260620 21:54:28.964610 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:54:28.965131 775 external_mini_cluster.cc:1428] Setting key 78e443174d2e5eb3b55846e734688139
I20260620 21:54:28.973774 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:28.967332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:41607 (local address 127.0.193.254:39821)
0620 21:54:28.967492 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:28.967500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:28.967514 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:54:28.968947 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:28.968951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:28.969005 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:28.969085 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:28.969979 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.970543 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.971751 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.971991 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.972108 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:28.973207 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:28.973211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:28.973213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:28.973227 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:28.973254 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:28.973261 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:28.973338 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:28.973549 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 21:54:28.973613 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:54:28.974761 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:28.966242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:39821 (local address 127.0.193.195:41607)
0620 21:54:28.967262 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:28.967321 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:28.968729 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:28.969253 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:28.969260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:28.969353 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:28.969834 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:28.969848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.970678 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.970682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:28.971618 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:28.971627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.972150 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.972154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:28.972322 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:28.972946 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:54:28.972947 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:28.972987 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:54:28.973053 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:28.973399 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:28.973406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:28.973410 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:54:28.973537 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 21:54:28.973843 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":175,"threads_started":1}
I20260620 21:54:28.976048 18820 heartbeater.cc:347] Connected to a master server at 127.0.193.254:39821
I20260620 21:54:28.976315 18820 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:28.976879 18820 heartbeater.cc:511] Master 127.0.193.254:39821 requested a full tablet report, sending...
I20260620 21:54:28.978039 18370 ts_manager.cc:194] Registered new tserver with Master: eb1cfed1cdc446fe8fa0d28a11452ed2 (127.0.193.195:34131)
I20260620 21:54:28.978803 18370 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.193.195:41607
I20260620 21:54:28.979861 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:54:28.986315 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:28.982094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.0.193.254:39821)
0620 21:54:28.982234 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:28.982239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:28.982402 (+ 163us) server_negotiation.cc:407] Connection header received
0620 21:54:28.982532 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:28.982534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:28.982574 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:28.982642 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:28.983023 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.983647 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.984566 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:28.984814 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:28.984968 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:28.985858 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:28.985862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:28.985865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:28.985881 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:28.985901 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:28.985909 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:28.985975 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:28.986053 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 21:54:28.986107 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:54:28.989714 18370 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:54:28.993214 18370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:54:29.009904 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:29.003972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.0.193.194:39029)
0620 21:54:29.005114 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:29.005145 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:29.005188 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:54:29.005267 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:29.005273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:29.005467 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:29.005623 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:29.005872 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.006913 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.008556 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.009120 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.009251 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:29.009487 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:29.009594 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 21:54:29.009712 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1032,"thread_start_us":88,"threads_started":1}
I20260620 21:54:29.010092 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:29.003894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:34131 (local address 127.0.0.1:60706)
0620 21:54:29.005063 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:29.005077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:29.005160 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:29.005797 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:29.005800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:29.005818 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:29.005908 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:29.005925 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.007312 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.007315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:29.008455 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:29.008465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.009703 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.009707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:29.009790 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:29.009822 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 21:54:29.009889 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 21:54:29.009955 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"mutex_wait_us":70,"thread_start_us":92,"threads_started":1}
I20260620 21:54:29.009953 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:29.003086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:39519 (local address 127.0.0.1:60024)
0620 21:54:29.003659 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:29.003724 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:29.003926 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:29.004732 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:29.004739 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:29.004763 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:29.004948 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:29.004958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.006633 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.006637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:29.008267 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:29.008280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.009438 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.009441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:29.009528 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:29.009551 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:54:29.009604 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 21:54:29.009756 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":69,"thread_start_us":148,"threads_started":1}
I20260620 21:54:29.010689 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:29.004085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.0.193.195:34131)
0620 21:54:29.004415 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:29.004429 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:29.005178 (+ 749us) server_negotiation.cc:407] Connection header received
0620 21:54:29.005294 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:29.005298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:29.005466 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:29.005623 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:29.006072 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.007164 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.008590 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.009347 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.009457 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:29.009640 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:29.010407 (+ 767us) server_negotiation.cc:299] Negotiation successful
0620 21:54:29.010527 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":117,"threads_started":1}
I20260620 21:54:29.010732 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:29.003379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:39029 (local address 127.0.0.1:52540)
0620 21:54:29.004710 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:29.004723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:29.004811 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:29.005618 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:29.005621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:29.005640 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:29.005732 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:29.005738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.007028 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.007033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:29.008446 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:29.008453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.009251 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.009255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:29.009391 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:29.009434 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 21:54:29.009564 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 21:54:29.009773 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1272,"mutex_wait_us":236,"thread_start_us":120,"threads_started":1}
I20260620 21:54:29.011153 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:29.003238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.0.193.193:39519)
0620 21:54:29.004253 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:29.004260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:29.004287 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:54:29.004378 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:29.004383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:29.004569 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:29.004745 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:29.005321 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.006500 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.008426 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.009268 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.009888 (+ 620us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:29.010087 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:29.010145 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 21:54:29.010344 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":121,"threads_started":1}
I20260620 21:54:29.013772 18623 tablet_service.cc:1467] Processing CreateTablet for tablet 35618f5b760d4e3eb9eb0a7894287ec7 (DEFAULT_TABLE table=test-table [id=8dba8868c3fc468d91912e348931069a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:29.013953 18754 tablet_service.cc:1467] Processing CreateTablet for tablet 35618f5b760d4e3eb9eb0a7894287ec7 (DEFAULT_TABLE table=test-table [id=8dba8868c3fc468d91912e348931069a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:29.014263 18492 tablet_service.cc:1467] Processing CreateTablet for tablet 35618f5b760d4e3eb9eb0a7894287ec7 (DEFAULT_TABLE table=test-table [id=8dba8868c3fc468d91912e348931069a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:29.014932 18623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35618f5b760d4e3eb9eb0a7894287ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:29.015038 18754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35618f5b760d4e3eb9eb0a7894287ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:29.015161 18492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35618f5b760d4e3eb9eb0a7894287ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:29.022173 18837 tablet_bootstrap.cc:492] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675: Bootstrap starting.
I20260620 21:54:29.023736 18839 tablet_bootstrap.cc:492] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb: Bootstrap starting.
I20260620 21:54:29.024583 18837 tablet_bootstrap.cc:654] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:29.025352 18837 log.cc:824] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:29.025660 18839 tablet_bootstrap.cc:654] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:29.026158 18838 tablet_bootstrap.cc:492] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2: Bootstrap starting.
I20260620 21:54:29.026538 18839 log.cc:824] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:29.027644 18837 tablet_bootstrap.cc:492] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675: No bootstrap required, opened a new log
I20260620 21:54:29.027853 18837 ts_tablet_manager.cc:1397] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 21:54:29.028472 18838 tablet_bootstrap.cc:654] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:29.028558 18839 tablet_bootstrap.cc:492] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb: No bootstrap required, opened a new log
I20260620 21:54:29.028757 18839 ts_tablet_manager.cc:1397] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 21:54:29.029294 18838 log.cc:824] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:29.031526 18838 tablet_bootstrap.cc:492] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2: No bootstrap required, opened a new log
I20260620 21:54:29.031733 18838 ts_tablet_manager.cc:1397] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:54:29.032397 18839 raft_consensus.cc:348] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be23c1ecfb3943debf80d9afecad8feb" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39519 } } peers { permanent_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 34131 } } peers { permanent_uuid: "52bf9c42dd864f89b122abc7c7403675" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39029 } }
I20260620 21:54:29.032402 18837 raft_consensus.cc:348] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be23c1ecfb3943debf80d9afecad8feb" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39519 } } peers { permanent_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 34131 } } peers { permanent_uuid: "52bf9c42dd864f89b122abc7c7403675" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39029 } }
I20260620 21:54:29.032647 18839 raft_consensus.cc:374] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:29.032651 18837 raft_consensus.cc:374] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:29.032701 18839 raft_consensus.cc:729] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be23c1ecfb3943debf80d9afecad8feb, State: Initialized, Role: FOLLOWER
I20260620 21:54:29.032702 18837 raft_consensus.cc:729] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bf9c42dd864f89b122abc7c7403675, State: Initialized, Role: FOLLOWER
I20260620 21:54:29.033186 18837 consensus_queue.cc:260] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be23c1ecfb3943debf80d9afecad8feb" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39519 } } peers { permanent_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 34131 } } peers { permanent_uuid: "52bf9c42dd864f89b122abc7c7403675" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39029 } }
I20260620 21:54:29.033408 18839 consensus_queue.cc:260] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be23c1ecfb3943debf80d9afecad8feb" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39519 } } peers { permanent_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 34131 } } peers { permanent_uuid: "52bf9c42dd864f89b122abc7c7403675" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39029 } }
I20260620 21:54:29.034293 18837 ts_tablet_manager.cc:1428] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:54:29.034512 18839 ts_tablet_manager.cc:1428] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:54:29.034694 18689 heartbeater.cc:503] Master 127.0.193.254:39821 was elected leader, sending a full tablet report...
I20260620 21:54:29.034842 18558 heartbeater.cc:503] Master 127.0.193.254:39821 was elected leader, sending a full tablet report...
I20260620 21:54:29.035893 18838 raft_consensus.cc:348] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be23c1ecfb3943debf80d9afecad8feb" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39519 } } peers { permanent_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 34131 } } peers { permanent_uuid: "52bf9c42dd864f89b122abc7c7403675" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39029 } }
I20260620 21:54:29.036196 18838 raft_consensus.cc:374] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:29.036264 18838 raft_consensus.cc:729] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb1cfed1cdc446fe8fa0d28a11452ed2, State: Initialized, Role: FOLLOWER
I20260620 21:54:29.036723 18838 consensus_queue.cc:260] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be23c1ecfb3943debf80d9afecad8feb" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39519 } } peers { permanent_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 34131 } } peers { permanent_uuid: "52bf9c42dd864f89b122abc7c7403675" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39029 } }
I20260620 21:54:29.037407 18820 heartbeater.cc:503] Master 127.0.193.254:39821 was elected leader, sending a full tablet report...
I20260620 21:54:29.037781 18838 ts_tablet_manager.cc:1428] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:54:29.038920 18845 raft_consensus.cc:482] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:54:29.039072 18845 raft_consensus.cc:504] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be23c1ecfb3943debf80d9afecad8feb" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39519 } } peers { permanent_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 34131 } } peers { permanent_uuid: "52bf9c42dd864f89b122abc7c7403675" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39029 } }
I20260620 21:54:29.041107 18845 leader_election.cc:288] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be23c1ecfb3943debf80d9afecad8feb (127.0.193.193:39519), 52bf9c42dd864f89b122abc7c7403675 (127.0.193.194:39029)
I20260620 21:54:29.044411 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:29.040889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:45621 (local address 127.0.193.194:39029)
0620 21:54:29.041003 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:29.041007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:29.041020 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:54:29.041113 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:29.041118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:29.041167 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:29.041238 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:29.041544 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.042064 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.043286 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.043758 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.043853 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:29.043894 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:29.044196 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 21:54:29.044279 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:54:29.044405 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:29.040549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:46233 (local address 127.0.193.193:39519)
0620 21:54:29.040662 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:29.040666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:29.040974 (+ 308us) server_negotiation.cc:407] Connection header received
0620 21:54:29.041166 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:29.041169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:29.041224 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:29.041289 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:29.041569 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.042142 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.043401 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.043864 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.043980 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:29.044028 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:29.044193 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 21:54:29.044272 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 21:54:29.044431 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:29.040506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:39519 (local address 127.0.193.195:46233)
0620 21:54:29.040891 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:29.040905 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:29.041039 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:29.041307 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:29.041310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:29.041325 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:29.041456 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:29.041461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.042264 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.042267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:29.043283 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:29.043293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.043999 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.044002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:29.044075 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:29.044092 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 21:54:29.044179 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 21:54:29.044294 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":29}
I20260620 21:54:29.044442 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:29.040530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:39029 (local address 127.0.193.195:45621)
0620 21:54:29.040914 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:29.040929 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:29.041013 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:29.041253 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:29.041255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:29.041269 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:29.041392 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:29.041398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.042175 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.042182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:29.043144 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:29.043152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.043877 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.043881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:29.044059 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:29.044089 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 21:54:29.044179 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 21:54:29.044296 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":22,"thread_start_us":135,"threads_started":1}
I20260620 21:54:29.045013 18512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35618f5b760d4e3eb9eb0a7894287ec7" candidate_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be23c1ecfb3943debf80d9afecad8feb" is_pre_election: true
I20260620 21:54:29.045192 18643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35618f5b760d4e3eb9eb0a7894287ec7" candidate_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bf9c42dd864f89b122abc7c7403675" is_pre_election: true
I20260620 21:54:29.045323 18512 raft_consensus.cc:2454] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1cfed1cdc446fe8fa0d28a11452ed2 in term 0.
I20260620 21:54:29.045475 18643 raft_consensus.cc:2454] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb1cfed1cdc446fe8fa0d28a11452ed2 in term 0.
I20260620 21:54:29.045814 18709 leader_election.cc:302] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be23c1ecfb3943debf80d9afecad8feb, eb1cfed1cdc446fe8fa0d28a11452ed2; no voters:
I20260620 21:54:29.046101 18845 raft_consensus.cc:2781] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:54:29.046187 18845 raft_consensus.cc:482] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:54:29.046229 18845 raft_consensus.cc:3037] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:29.047298 18845 raft_consensus.cc:504] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be23c1ecfb3943debf80d9afecad8feb" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39519 } } peers { permanent_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 34131 } } peers { permanent_uuid: "52bf9c42dd864f89b122abc7c7403675" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39029 } }
I20260620 21:54:29.047652 18845 leader_election.cc:288] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [CANDIDATE]: Term 1 election: Requested vote from peers be23c1ecfb3943debf80d9afecad8feb (127.0.193.193:39519), 52bf9c42dd864f89b122abc7c7403675 (127.0.193.194:39029)
I20260620 21:54:29.047973 18512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35618f5b760d4e3eb9eb0a7894287ec7" candidate_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be23c1ecfb3943debf80d9afecad8feb"
I20260620 21:54:29.048101 18643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35618f5b760d4e3eb9eb0a7894287ec7" candidate_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52bf9c42dd864f89b122abc7c7403675"
I20260620 21:54:29.048115 18512 raft_consensus.cc:3037] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:29.048238 18643 raft_consensus.cc:3037] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:29.049221 18512 raft_consensus.cc:2454] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1cfed1cdc446fe8fa0d28a11452ed2 in term 1.
I20260620 21:54:29.049263 18643 raft_consensus.cc:2454] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb1cfed1cdc446fe8fa0d28a11452ed2 in term 1.
I20260620 21:54:29.049533 18709 leader_election.cc:302] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be23c1ecfb3943debf80d9afecad8feb, eb1cfed1cdc446fe8fa0d28a11452ed2; no voters:
I20260620 21:54:29.049717 18845 raft_consensus.cc:2781] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:29.049964 18845 raft_consensus.cc:686] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [term 1 LEADER]: Becoming Leader. State: Replica: eb1cfed1cdc446fe8fa0d28a11452ed2, State: Running, Role: LEADER
I20260620 21:54:29.050302 18845 consensus_queue.cc:237] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be23c1ecfb3943debf80d9afecad8feb" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39519 } } peers { permanent_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 34131 } } peers { permanent_uuid: "52bf9c42dd864f89b122abc7c7403675" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39029 } }
I20260620 21:54:29.053720 18369 catalog_manager.cc:5495] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb1cfed1cdc446fe8fa0d28a11452ed2 (127.0.193.195). New cstate: current_term: 1 leader_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be23c1ecfb3943debf80d9afecad8feb" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb1cfed1cdc446fe8fa0d28a11452ed2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 34131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52bf9c42dd864f89b122abc7c7403675" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39029 } health_report { overall_health: UNKNOWN } } }
I20260620 21:54:29.075277 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:29.071668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.0.193.195:34131)
0620 21:54:29.071824 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:29.071828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:29.071839 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:54:29.071916 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:29.071918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:29.071979 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:29.072095 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:54:29.072347 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.072843 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.073737 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:29.073971 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:29.074079 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:29.074407 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:54:29.074844 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:54:29.074991 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:29.075024 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 21:54:29.075079 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:54:29.081612 18512 raft_consensus.cc:1261] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb [term 1 FOLLOWER]: Refusing update from remote peer eb1cfed1cdc446fe8fa0d28a11452ed2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:54:29.081617 18643 raft_consensus.cc:1261] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675 [term 1 FOLLOWER]: Refusing update from remote peer eb1cfed1cdc446fe8fa0d28a11452ed2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:54:29.082433 18847 consensus_queue.cc:1035] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52bf9c42dd864f89b122abc7c7403675" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:54:29.082739 18845 consensus_queue.cc:1035] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be23c1ecfb3943debf80d9afecad8feb" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 39519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:54:29.090863 18852 mvcc.cc:205] Tried to move back new op lower bound from 7299041153350496256 to 7299041153232859136. Current Snapshot: MvccSnapshot[applied={T|T < 7299041153350496256}]
I20260620 21:54:29.091207 18851 mvcc.cc:205] Tried to move back new op lower bound from 7299041153350496256 to 7299041153232859136. Current Snapshot: MvccSnapshot[applied={T|T < 7299041153350496256}]
I20260620 21:54:29.098974 18369 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:54:29.099141 18369 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:54:29.101958 18369 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7450e7ed7f444d418e44e848b7de2a00: Sending DeleteTablet for 3 replicas of tablet 35618f5b760d4e3eb9eb0a7894287ec7
I20260620 21:54:29.102986 18492 tablet_service.cc:1514] Processing DeleteTablet for tablet 35618f5b760d4e3eb9eb0a7894287ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:29 UTC) from {username='slave'} at 127.0.0.1:60024
I20260620 21:54:29.103646 18623 tablet_service.cc:1514] Processing DeleteTablet for tablet 35618f5b760d4e3eb9eb0a7894287ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:29 UTC) from {username='slave'} at 127.0.0.1:52540
I20260620 21:54:29.103718 18754 tablet_service.cc:1514] Processing DeleteTablet for tablet 35618f5b760d4e3eb9eb0a7894287ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:29 UTC) from {username='slave'} at 127.0.0.1:60706
I20260620 21:54:29.103763 18859 tablet_replica.cc:330] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb: stopping tablet replica
I20260620 21:54:29.104235 18861 tablet_replica.cc:330] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675: stopping tablet replica
I20260620 21:54:29.104403 18859 raft_consensus.cc:2229] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:54:29.104549 18861 raft_consensus.cc:2229] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:54:29.104897 18859 raft_consensus.cc:2258] T 35618f5b760d4e3eb9eb0a7894287ec7 P be23c1ecfb3943debf80d9afecad8feb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:54:29.104897 18861 raft_consensus.cc:2258] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:54:29.105235 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 18430
I20260620 21:54:29.108541 18861 ts_tablet_manager.cc:1905] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:54:29.111106 18860 tablet_replica.cc:330] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2: stopping tablet replica
I20260620 21:54:29.111613 18860 raft_consensus.cc:2229] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:54:29.111792 18861 ts_tablet_manager.cc:1918] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:54:29.111919 18861 log.cc:1196] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/35618f5b760d4e3eb9eb0a7894287ec7
I20260620 21:54:29.112262 18861 ts_tablet_manager.cc:1939] T 35618f5b760d4e3eb9eb0a7894287ec7 P 52bf9c42dd864f89b122abc7c7403675: Deleting consensus metadata
I20260620 21:54:29.112282 18860 raft_consensus.cc:2258] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 21:54:29.112916 18356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:39519 (error 108)
I20260620 21:54:29.113667 18354 catalog_manager.cc:4841] TS 52bf9c42dd864f89b122abc7c7403675 (127.0.193.194:39029): tablet 35618f5b760d4e3eb9eb0a7894287ec7 (table test-table [id=8dba8868c3fc468d91912e348931069a]) successfully deleted
I20260620 21:54:29.113770 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 18561
I20260620 21:54:29.115047 18860 ts_tablet_manager.cc:1905] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:54:29.117237 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:29.116747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:39519 (local address 127.0.0.1:60034)
0620 21:54:29.117001 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:29.117109 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 21:54:29.117738 18356 catalog_manager.cc:4584] TS be23c1ecfb3943debf80d9afecad8feb (127.0.193.193:39519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35618f5b760d4e3eb9eb0a7894287ec7: Network error: Client connection negotiation failed: client connection to 127.0.193.193:39519: connect: Connection refused (error 111)
I20260620 21:54:29.118427 18860 ts_tablet_manager.cc:1918] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:54:29.118551 18860 log.cc:1196] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/35618f5b760d4e3eb9eb0a7894287ec7
I20260620 21:54:29.118872 18860 ts_tablet_manager.cc:1939] T 35618f5b760d4e3eb9eb0a7894287ec7 P eb1cfed1cdc446fe8fa0d28a11452ed2: Deleting consensus metadata
I20260620 21:54:29.120311 18356 catalog_manager.cc:4841] TS eb1cfed1cdc446fe8fa0d28a11452ed2 (127.0.193.195:34131): tablet 35618f5b760d4e3eb9eb0a7894287ec7 (table test-table [id=8dba8868c3fc468d91912e348931069a]) successfully deleted
I20260620 21:54:29.121964 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 18692
I20260620 21:54:29.127722 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 18339
2026-06-20T21:54:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1103 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:29 dist-test-slave-kk5c krb5kdc[18865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:29 dist-test-slave-kk5c krb5kdc[18865](info): set up 2 sockets
Jun 20 21:54:29 dist-test-slave-kk5c krb5kdc[18865](info): commencing operation
krb5kdc: starting...
W20260620 21:54:30.191113 775 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:30 dist-test-slave-kk5c krb5kdc[18865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:54:30.323444 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:33187
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:40777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:33187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:30.449530 18878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:30.449884 18878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:30.449942 18878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:30.454516 18878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:30.454605 18878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:30.454625 18878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:30.454643 18878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:30.454659 18878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:30.460904 18878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:40777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:33187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:33187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:30.462157 18878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:30.463383 18878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:30.473691 18884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:30.473665 18883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:30.473665 18886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:30.473974 18878 server_base.cc:1034] running on GCE node
I20260620 21:54:30.474695 18878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:30.475915 18878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:30.477100 18878 hybrid_clock.cc:648] HybridClock initialized: now 1781992470477083 us; error 39 us; skew 500 ppm
Jun 20 21:54:30 dist-test-slave-kk5c krb5kdc[18865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:30.485747 18878 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:54:30.486891 18878 webserver.cc:460] Webserver started at http://127.0.193.254:35261/ using document root <none> and password file <none>
I20260620 21:54:30.487587 18878 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:30.487685 18878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:30.487927 18878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:30.490147 18878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "f5d335131de244c1a6d4a8fb5ad13ef5"
format_stamp: "Formatted at 2026-06-20 21:54:30 on dist-test-slave-kk5c"
server_key: "9b28b297986b1eec3abaff930875a1e4"
server_key_iv: "472d384d4be002f4583241698b3115d6"
server_key_version: "encryptionkey@0"
I20260620 21:54:30.490752 18878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "f5d335131de244c1a6d4a8fb5ad13ef5"
format_stamp: "Formatted at 2026-06-20 21:54:30 on dist-test-slave-kk5c"
server_key: "9b28b297986b1eec3abaff930875a1e4"
server_key_iv: "472d384d4be002f4583241698b3115d6"
server_key_version: "encryptionkey@0"
I20260620 21:54:30.494756 18878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:54:30.497503 18893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:30.498977 18878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:54:30.499154 18878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "f5d335131de244c1a6d4a8fb5ad13ef5"
format_stamp: "Formatted at 2026-06-20 21:54:30 on dist-test-slave-kk5c"
server_key: "9b28b297986b1eec3abaff930875a1e4"
server_key_iv: "472d384d4be002f4583241698b3115d6"
server_key_version: "encryptionkey@0"
I20260620 21:54:30.499364 18878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:30.516454 18878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:30.520094 18878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:30.520324 18878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:30.527772 18878 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:33187
I20260620 21:54:30.527822 18945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:33187 every 8 connection(s)
I20260620 21:54:30.528789 18878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:30.531096 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 18878
I20260620 21:54:30.531323 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:30.531678 775 external_mini_cluster.cc:1428] Setting key b10298bdb24134c61090d5b9225f8bce
I20260620 21:54:30.532241 18946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:30.540031 18946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5: Bootstrap starting.
Jun 20 21:54:30 dist-test-slave-kk5c krb5kdc[18865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:54:30.542878 18946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:30.543984 18946 log.cc:824] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:30.546206 18946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5: No bootstrap required, opened a new log
I20260620 21:54:30.548105 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:30.533397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.0.193.254:33187)
0620 21:54:30.534121 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:30.534137 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:30.534176 (+ 39us) server_negotiation.cc:407] Connection header received
0620 21:54:30.535110 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:30.535130 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:30.535585 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:30.535995 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:30.537112 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:30.538143 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:30.539063 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:30.539615 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:30.539795 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:30.542757 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:30.542763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:30.542778 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:30.542816 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:30.545009 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:30.545511 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:30.545516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:30.545521 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:30.545634 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:30.545989 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:30.545994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:30.545995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:30.546483 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:30.546693 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:30.547011 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 21:54:30.547377 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":207,"threads_started":1}
I20260620 21:54:30.550817 18946 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d335131de244c1a6d4a8fb5ad13ef5" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 33187 } }
I20260620 21:54:30.551088 18946 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:30.551178 18946 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d335131de244c1a6d4a8fb5ad13ef5, State: Initialized, Role: FOLLOWER
I20260620 21:54:30.551656 18946 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d335131de244c1a6d4a8fb5ad13ef5" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 33187 } }
I20260620 21:54:30.551813 18946 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:30.551906 18946 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:30.552034 18946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:30.553061 18946 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d335131de244c1a6d4a8fb5ad13ef5" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 33187 } }
I20260620 21:54:30.553433 18946 leader_election.cc:302] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5d335131de244c1a6d4a8fb5ad13ef5; no voters:
I20260620 21:54:30.553828 18946 leader_election.cc:288] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:30.554021 18951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:30.554281 18951 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [term 1 LEADER]: Becoming Leader. State: Replica: f5d335131de244c1a6d4a8fb5ad13ef5, State: Running, Role: LEADER
I20260620 21:54:30.554622 18951 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d335131de244c1a6d4a8fb5ad13ef5" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 33187 } }
I20260620 21:54:30.555171 18946 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:30.556282 18952 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5d335131de244c1a6d4a8fb5ad13ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d335131de244c1a6d4a8fb5ad13ef5" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 33187 } } }
I20260620 21:54:30.556420 18952 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:30.556762 18953 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5d335131de244c1a6d4a8fb5ad13ef5. Latest consensus state: current_term: 1 leader_uuid: "f5d335131de244c1a6d4a8fb5ad13ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d335131de244c1a6d4a8fb5ad13ef5" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 33187 } } }
I20260620 21:54:30.556866 18953 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5 [sys.catalog]: This master's current role is: LEADER
W20260620 21:54:30.560227 18966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:54:30.560326 18966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:54:30.560416 18960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:30.561177 18960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:30.566844 18960 catalog_manager.cc:1347] Generated new cluster ID: 20704538d19c42cd91cb3c17348695ad
I20260620 21:54:30.566969 18960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:30.599009 18960 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:30.600548 18960 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:30.616261 18960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5: Generated new TSK 0
I20260620 21:54:30.617426 18960 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:54:30.664603 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:33187
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:40777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:30.787616 18972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:30.787957 18972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:30.788044 18972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:30.792336 18972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:30.792450 18972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:30.792544 18972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:54:30.798352 18972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:40777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:33187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:30.799571 18972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:30.800634 18972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:30.810587 18980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:30.810591 18978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:30.810588 18977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:30.810640 18972 server_base.cc:1034] running on GCE node
I20260620 21:54:30.811340 18972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:30.812003 18972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:30.813202 18972 hybrid_clock.cc:648] HybridClock initialized: now 1781992470813180 us; error 32 us; skew 500 ppm
Jun 20 21:54:30 dist-test-slave-kk5c krb5kdc[18865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:30.816314 18972 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:54:30.817267 18972 webserver.cc:460] Webserver started at http://127.0.193.193:39581/ using document root <none> and password file <none>
I20260620 21:54:30.817853 18972 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:30.817950 18972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:30.818245 18972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:30.820397 18972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9"
format_stamp: "Formatted at 2026-06-20 21:54:30 on dist-test-slave-kk5c"
server_key: "6b54d11f70bb668d8d7b97b2fde0241c"
server_key_iv: "1ee3dec4fbc477ef5e46dd8d228a8bc1"
server_key_version: "encryptionkey@0"
I20260620 21:54:30.820993 18972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9"
format_stamp: "Formatted at 2026-06-20 21:54:30 on dist-test-slave-kk5c"
server_key: "6b54d11f70bb668d8d7b97b2fde0241c"
server_key_iv: "1ee3dec4fbc477ef5e46dd8d228a8bc1"
server_key_version: "encryptionkey@0"
I20260620 21:54:30.825002 18972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:54:30.827839 18987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:30.829281 18972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:30.829457 18972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9"
format_stamp: "Formatted at 2026-06-20 21:54:30 on dist-test-slave-kk5c"
server_key: "6b54d11f70bb668d8d7b97b2fde0241c"
server_key_iv: "1ee3dec4fbc477ef5e46dd8d228a8bc1"
server_key_version: "encryptionkey@0"
I20260620 21:54:30.829600 18972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:30.845952 18972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:30.849433 18972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:30.849649 18972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:30.850312 18972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:30.851164 18972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:30.851287 18972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:30.851359 18972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:30.851408 18972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:30.861333 18972 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:35141
I20260620 21:54:30.861356 19100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:35141 every 8 connection(s)
I20260620 21:54:30.862267 18972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:54:30.862648 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 18972
I20260620 21:54:30.862793 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:54:30.863157 775 external_mini_cluster.cc:1428] Setting key 417efb355a914ca7a751bd98d7ca0e36
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
Jun 20 21:54:30 dist-test-slave-kk5c krb5kdc[18865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
I20260620 21:54:30.877485 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:30.864450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:60411 (local address 127.0.193.254:33187)
0620 21:54:30.864599 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:30.864604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:30.865633 (+ 1029us) server_negotiation.cc:407] Connection header received
0620 21:54:30.866937 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:30.866940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:30.867005 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:30.867112 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:30.868371 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:30.868911 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:30.870106 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:30.870345 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:30.870457 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:30.873627 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:30.873630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:30.873633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:30.873659 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:30.875164 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:30.875836 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:30.875840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:30.875842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:30.875928 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:30.876306 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:30.876310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:30.876312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:30.876511 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:30.876627 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:30.877165 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 21:54:30.877313 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:54:30.878373 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:30.864982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:33187 (local address 127.0.193.193:60411)
0620 21:54:30.865486 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:30.865531 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:30.866726 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:30.867318 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:30.867325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:30.867695 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:30.868180 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:30.868194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:30.869053 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:30.869057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:30.869977 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:30.869987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:30.870510 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:30.870514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:30.870683 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:30.871655 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:54:30.871672 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:30.873424 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:30.875366 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:30.875374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:30.875391 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:30.875721 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:30.876041 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:30.876044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:30.876046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:30.876206 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:30.876624 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:30.876632 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:30.876907 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 21:54:30.877124 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 21:54:30.877484 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":132,"threads_started":1}
I20260620 21:54:30.879778 19101 heartbeater.cc:347] Connected to a master server at 127.0.193.254:33187
I20260620 21:54:30.880102 19101 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:30.880635 19101 heartbeater.cc:511] Master 127.0.193.254:33187 requested a full tablet report, sending...
I20260620 21:54:30.882349 18910 ts_manager.cc:194] Registered new tserver with Master: bc7785f7fdac4ef4be82c4c1ac2d3ba9 (127.0.193.193:35141)
I20260620 21:54:30.883621 18910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:60411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:54:30.896560 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:33187
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:40777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:31.024335 19106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:31.024643 19106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:31.024691 19106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:31.029052 19106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:31.029129 19106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:31.029193 19106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:54:31.035429 19106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:40777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:33187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:31.036598 19106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:31.037700 19106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 21:54:31.048252 19106 server_base.cc:1034] running on GCE node
W20260620 21:54:31.048240 19111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:31.048269 19112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:31.048259 19114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:31.049052 19106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:31.049721 19106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:31.050932 19106 hybrid_clock.cc:648] HybridClock initialized: now 1781992471050900 us; error 48 us; skew 500 ppm
Jun 20 21:54:31 dist-test-slave-kk5c krb5kdc[18865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:31.053910 19106 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:54:31.054901 19106 webserver.cc:460] Webserver started at http://127.0.193.194:44509/ using document root <none> and password file <none>
I20260620 21:54:31.055626 19106 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:31.055722 19106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:31.056010 19106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:31.058375 19106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "2eec02fbf1394ffeab54e66ba6155206"
format_stamp: "Formatted at 2026-06-20 21:54:31 on dist-test-slave-kk5c"
server_key: "d08b787d01792d8e5353aa02ce4c971e"
server_key_iv: "7498cfb67b6ff2e280d9c8d60c149c76"
server_key_version: "encryptionkey@0"
I20260620 21:54:31.058976 19106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "2eec02fbf1394ffeab54e66ba6155206"
format_stamp: "Formatted at 2026-06-20 21:54:31 on dist-test-slave-kk5c"
server_key: "d08b787d01792d8e5353aa02ce4c971e"
server_key_iv: "7498cfb67b6ff2e280d9c8d60c149c76"
server_key_version: "encryptionkey@0"
I20260620 21:54:31.063038 19106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 21:54:31.065979 19121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:31.067678 19106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:31.067869 19106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "2eec02fbf1394ffeab54e66ba6155206"
format_stamp: "Formatted at 2026-06-20 21:54:31 on dist-test-slave-kk5c"
server_key: "d08b787d01792d8e5353aa02ce4c971e"
server_key_iv: "7498cfb67b6ff2e280d9c8d60c149c76"
server_key_version: "encryptionkey@0"
I20260620 21:54:31.068019 19106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:31.088876 19106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:31.093132 19106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:31.093477 19106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:31.094173 19106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:31.095080 19106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:31.095155 19106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:31.095253 19106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:31.095299 19106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:31.110821 19106 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:41103
I20260620 21:54:31.111258 19234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:41103 every 8 connection(s)
I20260620 21:54:31.112238 19106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:54:31.114902 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 19106
I20260620 21:54:31.115046 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:54:31.115410 775 external_mini_cluster.cc:1428] Setting key faa152572b5307a479798028e466bd34
Jun 20 21:54:31 dist-test-slave-kk5c krb5kdc[18865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
I20260620 21:54:31.134310 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.114775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:48707 (local address 127.0.193.254:33187)
0620 21:54:31.114971 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:31.114976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:31.115943 (+ 967us) server_negotiation.cc:407] Connection header received
0620 21:54:31.117101 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:31.117104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:31.117174 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:31.117258 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:31.119146 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.120198 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.122708 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.122970 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.123071 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.127033 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:31.127037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:31.127039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:31.127072 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:31.130153 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:31.131737 (+ 1584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:31.131746 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:31.131747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:31.131846 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:31.132431 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:31.132434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:31.132435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:31.132667 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:31.132752 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:31.133851 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0620 21:54:31.134039 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 21:54:31.135413 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.115141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:33187 (local address 127.0.193.194:48707)
0620 21:54:31.115747 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:31.115812 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:31.116860 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:31.117480 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:31.117492 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:31.118105 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:31.118911 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:31.118933 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.120444 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.120447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:31.122508 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:31.122527 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.123168 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.123176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:31.123479 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.124359 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:54:31.124390 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:31.126711 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:31.130386 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:31.130393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:31.130411 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:31.131552 (+ 1141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:31.131988 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:31.131991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:31.131992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:31.132172 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:31.132928 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:31.132934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:31.133398 (+ 464us) client_negotiation.cc:769] Sending connection context
0620 21:54:31.133671 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 21:54:31.134158 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":121,"threads_started":1}
I20260620 21:54:31.137454 19235 heartbeater.cc:347] Connected to a master server at 127.0.193.254:33187
I20260620 21:54:31.137830 19235 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:31.138447 19235 heartbeater.cc:511] Master 127.0.193.254:33187 requested a full tablet report, sending...
I20260620 21:54:31.140026 18910 ts_manager.cc:194] Registered new tserver with Master: 2eec02fbf1394ffeab54e66ba6155206 (127.0.193.194:41103)
I20260620 21:54:31.141057 18910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:48707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:54:31.155328 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:33187
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:40777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:31.271931 19240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:31.272357 19240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:31.272415 19240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:31.277021 19240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:31.277118 19240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:31.277186 19240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:54:31.283103 19240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:40777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:33187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:31.284652 19240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:31.285825 19240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:31.297601 19248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:31.297637 19246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:31.297601 19245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:31.299252 19240 server_base.cc:1034] running on GCE node
I20260620 21:54:31.299887 19240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:31.300964 19240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:31.302594 19240 hybrid_clock.cc:648] HybridClock initialized: now 1781992471302496 us; error 111 us; skew 500 ppm
Jun 20 21:54:31 dist-test-slave-kk5c krb5kdc[18865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:31.306710 19240 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:54:31.308104 19240 webserver.cc:460] Webserver started at http://127.0.193.195:44565/ using document root <none> and password file <none>
I20260620 21:54:31.308768 19240 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:31.308861 19240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:31.309109 19240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:31.311584 19240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "68b9da4364614f668eca8faab8f69bea"
format_stamp: "Formatted at 2026-06-20 21:54:31 on dist-test-slave-kk5c"
server_key: "b4aaea02204097602212503e29c425a6"
server_key_iv: "2e0a699935563b313be4fa5860f6066f"
server_key_version: "encryptionkey@0"
I20260620 21:54:31.312271 19240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "68b9da4364614f668eca8faab8f69bea"
format_stamp: "Formatted at 2026-06-20 21:54:31 on dist-test-slave-kk5c"
server_key: "b4aaea02204097602212503e29c425a6"
server_key_iv: "2e0a699935563b313be4fa5860f6066f"
server_key_version: "encryptionkey@0"
I20260620 21:54:31.316776 19240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:54:31.320020 19255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:31.321489 19240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:54:31.321611 19240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "68b9da4364614f668eca8faab8f69bea"
format_stamp: "Formatted at 2026-06-20 21:54:31 on dist-test-slave-kk5c"
server_key: "b4aaea02204097602212503e29c425a6"
server_key_iv: "2e0a699935563b313be4fa5860f6066f"
server_key_version: "encryptionkey@0"
I20260620 21:54:31.321775 19240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:31.335736 19240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:31.339756 19240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:31.339989 19240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:31.340639 19240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:31.341538 19240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:31.341589 19240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:31.341658 19240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:31.341691 19240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:31.351922 19240 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:36567
I20260620 21:54:31.351938 19368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:36567 every 8 connection(s)
I20260620 21:54:31.352926 19240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:54:31.363021 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 19240
I20260620 21:54:31.363227 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:54:31.363540 775 external_mini_cluster.cc:1428] Setting key 9e80c0280a6abd4a08387a1403ee0f8c
Jun 20 21:54:31 dist-test-slave-kk5c krb5kdc[18865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:54:31.368252 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.355039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:39171 (local address 127.0.193.254:33187)
0620 21:54:31.355253 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:31.355258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:31.356134 (+ 876us) server_negotiation.cc:407] Connection header received
0620 21:54:31.357631 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:31.357634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:31.357687 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:31.357773 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:31.359157 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.359741 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.360926 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.361147 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.361232 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.364504 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:31.364507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:31.364510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:31.364547 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:31.366158 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:31.366687 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:31.366690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:31.366691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:31.366764 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:31.367135 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:31.367137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:31.367139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:31.367387 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:31.367498 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:31.368025 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 21:54:31.368131 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:54:31.369196 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.355356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:33187 (local address 127.0.193.195:39171)
0620 21:54:31.355947 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:31.356005 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:31.357341 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:31.358030 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:31.358037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:31.358486 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:31.358985 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:31.358999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.359882 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.359885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:31.360795 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:31.360805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.361291 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.361294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:31.361487 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.362338 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:54:31.362362 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:31.364321 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:31.366264 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:31.366269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:31.366282 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:31.366594 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:31.366857 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:31.366860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:31.366862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:31.367040 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:31.367515 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:31.367521 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:31.367777 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 21:54:31.367975 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 21:54:31.368252 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":163,"threads_started":1}
I20260620 21:54:31.370429 19369 heartbeater.cc:347] Connected to a master server at 127.0.193.254:33187
I20260620 21:54:31.370695 19369 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:31.371227 19369 heartbeater.cc:511] Master 127.0.193.254:33187 requested a full tablet report, sending...
I20260620 21:54:31.372349 18910 ts_manager.cc:194] Registered new tserver with Master: 68b9da4364614f668eca8faab8f69bea (127.0.193.195:36567)
I20260620 21:54:31.373067 18910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:39171
I20260620 21:54:31.377924 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:54:31.388373 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.379955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34780 (local address 127.0.193.254:33187)
0620 21:54:31.380100 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:31.380106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:31.380272 (+ 166us) server_negotiation.cc:407] Connection header received
0620 21:54:31.380479 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:31.380487 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:31.380528 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:31.380608 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:31.381332 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.381853 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.382723 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.382949 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.383058 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.384978 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:31.384983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:31.384986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:31.385020 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:31.386456 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:31.387011 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:31.387014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:31.387017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:31.387098 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:31.387487 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:31.387490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:31.387492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:31.387711 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:31.387931 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:31.388113 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 21:54:31.388210 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:54:31.391741 18910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:54:31.394886 18910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:54:31.411262 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.403963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:35141 (local address 127.0.0.1:37308)
0620 21:54:31.404766 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:31.404821 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:31.405039 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:31.405895 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:31.406007 (+ 112us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:31.406055 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:31.406244 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:31.406252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.407861 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.407864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:31.409374 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:31.409386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.410511 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.410513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:31.410682 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.410747 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 21:54:31.410901 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 21:54:31.411039 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":122,"thread_start_us":88,"threads_started":1}
I20260620 21:54:31.411291 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.404137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.0.193.193:35141)
0620 21:54:31.404492 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:31.404500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:31.404923 (+ 423us) server_negotiation.cc:407] Connection header received
0620 21:54:31.405248 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:31.405258 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:31.405519 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:31.405657 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:31.406433 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.407701 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.409519 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.410240 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.410309 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.410429 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:31.410960 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 21:54:31.411081 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":95,"threads_started":1}
I20260620 21:54:31.411388 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.404733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:36567 (local address 127.0.0.1:52130)
0620 21:54:31.405552 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:31.405569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:31.405659 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:31.406174 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:31.406177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:31.406196 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:31.406333 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:31.406342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.408701 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.408705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:31.409506 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:31.409515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.410617 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.410620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:31.410715 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.410758 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 21:54:31.410866 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 21:54:31.411039 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":53,"thread_start_us":106,"threads_started":1}
I20260620 21:54:31.411505 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.405171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.0.193.195:36567)
0620 21:54:31.405568 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:31.405579 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:31.405631 (+ 52us) server_negotiation.cc:407] Connection header received
0620 21:54:31.405851 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:31.405856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:31.406006 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:31.406200 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:31.407294 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.408535 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.409642 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.410485 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.410605 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.410757 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:31.411222 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 21:54:31.411356 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":194,"threads_started":1}
I20260620 21:54:31.411293 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.404173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:41103 (local address 127.0.0.1:35118)
0620 21:54:31.405245 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:31.405275 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:31.405356 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:31.406628 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:31.406631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:31.406650 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:31.406739 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:31.406745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.408437 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.408440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:31.409678 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:31.409688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.410837 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.410841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:31.410948 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.410974 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 21:54:31.411044 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 21:54:31.411112 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":61,"thread_start_us":103,"threads_started":1}
I20260620 21:54:31.411959 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.404813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.0.193.194:41103)
0620 21:54:31.406078 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:31.406096 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:31.406123 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:54:31.406205 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:31.406210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:31.406436 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:31.406565 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:31.406902 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.408039 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.409795 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.410608 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.410711 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.410831 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:31.411623 (+ 792us) server_negotiation.cc:299] Negotiation successful
0620 21:54:31.411736 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006,"thread_start_us":107,"threads_started":1}
I20260620 21:54:31.414929 19035 tablet_service.cc:1467] Processing CreateTablet for tablet d03371ed88f24513b709907bbbc9046b (DEFAULT_TABLE table=test-table [id=301a26e0c7e7454cb35350ba6e2577d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:31.414916 19303 tablet_service.cc:1467] Processing CreateTablet for tablet d03371ed88f24513b709907bbbc9046b (DEFAULT_TABLE table=test-table [id=301a26e0c7e7454cb35350ba6e2577d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:31.415118 19169 tablet_service.cc:1467] Processing CreateTablet for tablet d03371ed88f24513b709907bbbc9046b (DEFAULT_TABLE table=test-table [id=301a26e0c7e7454cb35350ba6e2577d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:31.416150 19169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d03371ed88f24513b709907bbbc9046b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:31.416163 19035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d03371ed88f24513b709907bbbc9046b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:31.416174 19303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d03371ed88f24513b709907bbbc9046b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:31.423570 19386 tablet_bootstrap.cc:492] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206: Bootstrap starting.
I20260620 21:54:31.424485 19388 tablet_bootstrap.cc:492] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9: Bootstrap starting.
I20260620 21:54:31.424767 19387 tablet_bootstrap.cc:492] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea: Bootstrap starting.
I20260620 21:54:31.426383 19386 tablet_bootstrap.cc:654] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:31.426678 19388 tablet_bootstrap.cc:654] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:31.427007 19387 tablet_bootstrap.cc:654] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:31.427553 19388 log.cc:824] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:31.427845 19386 log.cc:824] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:31.428300 19387 log.cc:824] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:31.430081 19388 tablet_bootstrap.cc:492] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9: No bootstrap required, opened a new log
I20260620 21:54:31.430169 19386 tablet_bootstrap.cc:492] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206: No bootstrap required, opened a new log
I20260620 21:54:31.430169 19387 tablet_bootstrap.cc:492] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea: No bootstrap required, opened a new log
I20260620 21:54:31.430330 19388 ts_tablet_manager.cc:1397] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:54:31.430374 19387 ts_tablet_manager.cc:1397] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:54:31.430374 19386 ts_tablet_manager.cc:1397] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 21:54:31.434114 19388 raft_consensus.cc:348] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35141 } } peers { permanent_uuid: "2eec02fbf1394ffeab54e66ba6155206" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41103 } } peers { permanent_uuid: "68b9da4364614f668eca8faab8f69bea" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 36567 } }
I20260620 21:54:31.434469 19388 raft_consensus.cc:374] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:31.434530 19388 raft_consensus.cc:729] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc7785f7fdac4ef4be82c4c1ac2d3ba9, State: Initialized, Role: FOLLOWER
I20260620 21:54:31.434711 19387 raft_consensus.cc:348] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35141 } } peers { permanent_uuid: "2eec02fbf1394ffeab54e66ba6155206" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41103 } } peers { permanent_uuid: "68b9da4364614f668eca8faab8f69bea" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 36567 } }
I20260620 21:54:31.434711 19386 raft_consensus.cc:348] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35141 } } peers { permanent_uuid: "2eec02fbf1394ffeab54e66ba6155206" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41103 } } peers { permanent_uuid: "68b9da4364614f668eca8faab8f69bea" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 36567 } }
I20260620 21:54:31.435019 19386 raft_consensus.cc:374] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:31.435019 19387 raft_consensus.cc:374] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:31.435082 19386 raft_consensus.cc:729] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eec02fbf1394ffeab54e66ba6155206, State: Initialized, Role: FOLLOWER
I20260620 21:54:31.435082 19387 raft_consensus.cc:729] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68b9da4364614f668eca8faab8f69bea, State: Initialized, Role: FOLLOWER
I20260620 21:54:31.435019 19388 consensus_queue.cc:260] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35141 } } peers { permanent_uuid: "2eec02fbf1394ffeab54e66ba6155206" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41103 } } peers { permanent_uuid: "68b9da4364614f668eca8faab8f69bea" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 36567 } }
I20260620 21:54:31.435581 19387 consensus_queue.cc:260] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35141 } } peers { permanent_uuid: "2eec02fbf1394ffeab54e66ba6155206" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41103 } } peers { permanent_uuid: "68b9da4364614f668eca8faab8f69bea" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 36567 } }
I20260620 21:54:31.435581 19386 consensus_queue.cc:260] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35141 } } peers { permanent_uuid: "2eec02fbf1394ffeab54e66ba6155206" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41103 } } peers { permanent_uuid: "68b9da4364614f668eca8faab8f69bea" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 36567 } }
I20260620 21:54:31.436619 19387 ts_tablet_manager.cc:1428] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:54:31.436619 19386 ts_tablet_manager.cc:1428] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:54:31.436621 19388 ts_tablet_manager.cc:1428] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 21:54:31.437033 19101 heartbeater.cc:503] Master 127.0.193.254:33187 was elected leader, sending a full tablet report...
I20260620 21:54:31.437299 19235 heartbeater.cc:503] Master 127.0.193.254:33187 was elected leader, sending a full tablet report...
I20260620 21:54:31.437446 19369 heartbeater.cc:503] Master 127.0.193.254:33187 was elected leader, sending a full tablet report...
W20260620 21:54:31.603909 19370 tablet.cc:2369] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:54:31.613722 19236 tablet.cc:2369] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:54:31.613974 19102 tablet.cc:2369] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:54:31.655107 19392 raft_consensus.cc:482] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:54:31.655426 19392 raft_consensus.cc:504] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35141 } } peers { permanent_uuid: "2eec02fbf1394ffeab54e66ba6155206" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41103 } } peers { permanent_uuid: "68b9da4364614f668eca8faab8f69bea" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 36567 } }
I20260620 21:54:31.656886 19392 leader_election.cc:288] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2eec02fbf1394ffeab54e66ba6155206 (127.0.193.194:41103), 68b9da4364614f668eca8faab8f69bea (127.0.193.195:36567)
I20260620 21:54:31.660928 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.657163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:55311 (local address 127.0.193.194:41103)
0620 21:54:31.657313 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:31.657320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:31.657649 (+ 329us) server_negotiation.cc:407] Connection header received
0620 21:54:31.657778 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:31.657784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:31.657833 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:31.657919 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:31.658242 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.658794 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.659960 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.660355 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.660487 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.660543 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:31.660730 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 21:54:31.660803 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:54:31.660960 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.657078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:41103 (local address 127.0.193.193:55311)
0620 21:54:31.657525 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:31.657547 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:31.657653 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:31.657932 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:31.657934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:31.657949 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:31.658126 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:31.658131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.658923 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.658927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:31.659824 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:31.659833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.660488 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.660491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:31.660587 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.660617 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 21:54:31.660701 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 21:54:31.660776 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":46,"spinlock_wait_cycles":26880,"thread_start_us":144,"threads_started":1}
I20260620 21:54:31.661123 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.657322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:36031 (local address 127.0.193.195:36567)
0620 21:54:31.657460 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:31.657465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:31.657693 (+ 228us) server_negotiation.cc:407] Connection header received
0620 21:54:31.657811 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:31.657814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:31.657856 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:31.657944 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:31.658287 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.658851 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.660104 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.660541 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.660682 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.660737 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:31.660931 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 21:54:31.661003 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:54:31.661149 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.657160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:36567 (local address 127.0.193.193:36031)
0620 21:54:31.657522 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:31.657542 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:31.657692 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:31.657967 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:31.657970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:31.658016 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:31.658155 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:31.658161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.658973 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.658980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:31.659961 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:31.659971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.660672 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.660675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:31.660791 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.660816 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 21:54:31.660912 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 21:54:31.660986 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":42,"spinlock_wait_cycles":4096,"thread_start_us":128,"threads_started":1}
I20260620 21:54:31.661594 19189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03371ed88f24513b709907bbbc9046b" candidate_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eec02fbf1394ffeab54e66ba6155206" is_pre_election: true
I20260620 21:54:31.661926 19189 raft_consensus.cc:2454] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc7785f7fdac4ef4be82c4c1ac2d3ba9 in term 0.
I20260620 21:54:31.661885 19323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03371ed88f24513b709907bbbc9046b" candidate_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68b9da4364614f668eca8faab8f69bea" is_pre_election: true
I20260620 21:54:31.662236 19323 raft_consensus.cc:2454] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc7785f7fdac4ef4be82c4c1ac2d3ba9 in term 0.
I20260620 21:54:31.662600 18989 leader_election.cc:302] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eec02fbf1394ffeab54e66ba6155206, bc7785f7fdac4ef4be82c4c1ac2d3ba9; no voters:
I20260620 21:54:31.662920 19392 raft_consensus.cc:2781] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:54:31.663013 19392 raft_consensus.cc:482] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:54:31.663057 19392 raft_consensus.cc:3037] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:31.664206 19392 raft_consensus.cc:504] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35141 } } peers { permanent_uuid: "2eec02fbf1394ffeab54e66ba6155206" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41103 } } peers { permanent_uuid: "68b9da4364614f668eca8faab8f69bea" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 36567 } }
I20260620 21:54:31.664590 19392 leader_election.cc:288] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [CANDIDATE]: Term 1 election: Requested vote from peers 2eec02fbf1394ffeab54e66ba6155206 (127.0.193.194:41103), 68b9da4364614f668eca8faab8f69bea (127.0.193.195:36567)
I20260620 21:54:31.665009 19189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03371ed88f24513b709907bbbc9046b" candidate_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eec02fbf1394ffeab54e66ba6155206"
I20260620 21:54:31.665148 19189 raft_consensus.cc:3037] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:31.665162 19323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03371ed88f24513b709907bbbc9046b" candidate_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68b9da4364614f668eca8faab8f69bea"
I20260620 21:54:31.665318 19323 raft_consensus.cc:3037] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:31.666371 19189 raft_consensus.cc:2454] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc7785f7fdac4ef4be82c4c1ac2d3ba9 in term 1.
I20260620 21:54:31.666532 19323 raft_consensus.cc:2454] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc7785f7fdac4ef4be82c4c1ac2d3ba9 in term 1.
I20260620 21:54:31.666714 18989 leader_election.cc:302] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eec02fbf1394ffeab54e66ba6155206, bc7785f7fdac4ef4be82c4c1ac2d3ba9; no voters:
I20260620 21:54:31.666896 19392 raft_consensus.cc:2781] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:31.667104 19392 raft_consensus.cc:686] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [term 1 LEADER]: Becoming Leader. State: Replica: bc7785f7fdac4ef4be82c4c1ac2d3ba9, State: Running, Role: LEADER
I20260620 21:54:31.667479 19392 consensus_queue.cc:237] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35141 } } peers { permanent_uuid: "2eec02fbf1394ffeab54e66ba6155206" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41103 } } peers { permanent_uuid: "68b9da4364614f668eca8faab8f69bea" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 36567 } }
I20260620 21:54:31.670670 18910 catalog_manager.cc:5495] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc7785f7fdac4ef4be82c4c1ac2d3ba9 (127.0.193.193). New cstate: current_term: 1 leader_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc7785f7fdac4ef4be82c4c1ac2d3ba9" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 35141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2eec02fbf1394ffeab54e66ba6155206" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68b9da4364614f668eca8faab8f69bea" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 36567 } health_report { overall_health: UNKNOWN } } }
I20260620 21:54:31.710237 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.706205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.0.193.193:35141)
0620 21:54:31.706389 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:31.706394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:31.706409 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:54:31.706492 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:31.706495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:31.706566 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:31.706667 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:54:31.706908 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.707555 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.708532 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:31.708863 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:31.708964 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:31.709294 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:54:31.709706 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:54:31.709869 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:31.709988 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 21:54:31.710052 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 21:54:31.716665 19323 raft_consensus.cc:1261] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea [term 1 FOLLOWER]: Refusing update from remote peer bc7785f7fdac4ef4be82c4c1ac2d3ba9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:54:31.716663 19189 raft_consensus.cc:1261] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206 [term 1 FOLLOWER]: Refusing update from remote peer bc7785f7fdac4ef4be82c4c1ac2d3ba9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:54:31.717494 19397 consensus_queue.cc:1035] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68b9da4364614f668eca8faab8f69bea" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 36567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:54:31.717684 19392 consensus_queue.cc:1035] T d03371ed88f24513b709907bbbc9046b P bc7785f7fdac4ef4be82c4c1ac2d3ba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eec02fbf1394ffeab54e66ba6155206" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:54:31.725865 19402 mvcc.cc:205] Tried to move back new op lower bound from 7299041164142432256 to 7299041163951964160. Current Snapshot: MvccSnapshot[applied={T|T < 7299041164142432256}]
I20260620 21:54:31.726096 19401 mvcc.cc:205] Tried to move back new op lower bound from 7299041164142432256 to 7299041163951964160. Current Snapshot: MvccSnapshot[applied={T|T < 7299041164142432256}]
I20260620 21:54:31.735488 18908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:54:31.735709 18908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:54:31.738631 18908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5d335131de244c1a6d4a8fb5ad13ef5: Sending DeleteTablet for 3 replicas of tablet d03371ed88f24513b709907bbbc9046b
I20260620 21:54:31.739742 19303 tablet_service.cc:1514] Processing DeleteTablet for tablet d03371ed88f24513b709907bbbc9046b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:31 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:52130
I20260620 21:54:31.739732 19169 tablet_service.cc:1514] Processing DeleteTablet for tablet d03371ed88f24513b709907bbbc9046b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:31 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:35118
I20260620 21:54:31.740298 19411 tablet_replica.cc:330] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea: stopping tablet replica
I20260620 21:54:31.740321 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 18972
I20260620 21:54:31.740648 19411 raft_consensus.cc:2229] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:54:31.741037 19411 raft_consensus.cc:2258] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:54:31.741398 19410 tablet_replica.cc:330] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206: stopping tablet replica
I20260620 21:54:31.741696 19410 raft_consensus.cc:2229] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:54:31.741994 19410 raft_consensus.cc:2258] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:54:31.743269 19411 ts_tablet_manager.cc:1905] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:54:31.743630 19410 ts_tablet_manager.cc:1905] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:54:31.745769 19411 ts_tablet_manager.cc:1918] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:54:31.745887 19411 log.cc:1196] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/d03371ed88f24513b709907bbbc9046b
I20260620 21:54:31.746227 19411 ts_tablet_manager.cc:1939] T d03371ed88f24513b709907bbbc9046b P 68b9da4364614f668eca8faab8f69bea: Deleting consensus metadata
I20260620 21:54:31.746932 19410 ts_tablet_manager.cc:1918] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:54:31.747035 19410 log.cc:1196] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/d03371ed88f24513b709907bbbc9046b
I20260620 21:54:31.747382 19410 ts_tablet_manager.cc:1939] T d03371ed88f24513b709907bbbc9046b P 2eec02fbf1394ffeab54e66ba6155206: Deleting consensus metadata
I20260620 21:54:31.747499 18897 catalog_manager.cc:4841] TS 68b9da4364614f668eca8faab8f69bea (127.0.193.195:36567): tablet d03371ed88f24513b709907bbbc9046b (table test-table [id=301a26e0c7e7454cb35350ba6e2577d1]) successfully deleted
I20260620 21:54:31.748565 18895 catalog_manager.cc:4841] TS 2eec02fbf1394ffeab54e66ba6155206 (127.0.193.194:41103): tablet d03371ed88f24513b709907bbbc9046b (table test-table [id=301a26e0c7e7454cb35350ba6e2577d1]) successfully deleted
W20260620 21:54:31.748921 18897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:35141 (error 108)
I20260620 21:54:31.749188 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 19106
I20260620 21:54:31.750190 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:31.749872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:35141 (local address 127.0.0.1:37322)
0620 21:54:31.750040 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:31.750117 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 21:54:31.750541 18897 catalog_manager.cc:4584] TS bc7785f7fdac4ef4be82c4c1ac2d3ba9 (127.0.193.193:35141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d03371ed88f24513b709907bbbc9046b: Network error: Client connection negotiation failed: client connection to 127.0.193.193:35141: connect: Connection refused (error 111)
I20260620 21:54:31.756285 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 19240
I20260620 21:54:31.762090 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 18878
2026-06-20T21:54:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2634 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T21:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:31Z Disabled control of system clock
I20260620 21:54:31.794198 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:41577
--webserver_interface=127.0.193.254
--webserver_port=0
--builtin_ntp_servers=127.0.193.212:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:41577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:54:31.915397 19416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:31.915712 19416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:31.915751 19416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:31.919960 19416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:31.920040 19416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:31.920055 19416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:31.920073 19416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:31.920089 19416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:31.925429 19416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:41577
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:41577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:31.926709 19416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:31.928227 19416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:31.937474 19422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:31.937515 19416 server_base.cc:1034] running on GCE node
W20260620 21:54:31.937701 19424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:31.937521 19421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:31.938357 19416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:31.939683 19416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:31.940908 19416 hybrid_clock.cc:648] HybridClock initialized: now 1781992471940885 us; error 37 us; skew 500 ppm
I20260620 21:54:31.943023 19416 webserver.cc:460] Webserver started at http://127.0.193.254:41669/ using document root <none> and password file <none>
I20260620 21:54:31.943687 19416 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:31.943783 19416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:31.944024 19416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:31.946211 19416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "5b375279556e4b90bcde4736e90ff45a"
format_stamp: "Formatted at 2026-06-20 21:54:31 on dist-test-slave-kk5c"
server_key: "0007736c0c4d64745a5b8e568b9c7ee3"
server_key_iv: "d0a2e4f851a6da43227635deb389b76d"
server_key_version: "encryptionkey@0"
I20260620 21:54:31.946817 19416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "5b375279556e4b90bcde4736e90ff45a"
format_stamp: "Formatted at 2026-06-20 21:54:31 on dist-test-slave-kk5c"
server_key: "0007736c0c4d64745a5b8e568b9c7ee3"
server_key_iv: "d0a2e4f851a6da43227635deb389b76d"
server_key_version: "encryptionkey@0"
I20260620 21:54:31.950791 19416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:54:31.953513 19430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:31.954946 19416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:54:31.955119 19416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "5b375279556e4b90bcde4736e90ff45a"
format_stamp: "Formatted at 2026-06-20 21:54:31 on dist-test-slave-kk5c"
server_key: "0007736c0c4d64745a5b8e568b9c7ee3"
server_key_iv: "d0a2e4f851a6da43227635deb389b76d"
server_key_version: "encryptionkey@0"
I20260620 21:54:31.955309 19416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:31.986443 19416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:31.987169 19416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:31.987442 19416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:31.995152 19416 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:41577
I20260620 21:54:31.995162 19482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:41577 every 8 connection(s)
I20260620 21:54:31.996354 19416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:31.999722 19483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:32.001940 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 19416
I20260620 21:54:32.002130 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:32.002441 775 external_mini_cluster.cc:1428] Setting key 2a2d594626674e5e7071a47ca1b654c9
I20260620 21:54:32.006656 19483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a: Bootstrap starting.
I20260620 21:54:32.009570 19483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:32.010596 19483 log.cc:824] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:32.013068 19483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a: No bootstrap required, opened a new log
I20260620 21:54:32.013283 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.004207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.0.193.254:41577)
0620 21:54:32.004927 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:32.004944 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:32.004984 (+ 40us) server_negotiation.cc:407] Connection header received
0620 21:54:32.005699 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:32.005718 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:32.006154 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:32.006472 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:32.006923 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.008079 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.009366 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.010002 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.010136 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.011173 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:32.011180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:32.011222 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:32.011742 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:32.011804 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:32.011826 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:32.011960 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:32.012133 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 21:54:32.012434 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":166,"threads_started":1}
I20260620 21:54:32.017537 19483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b375279556e4b90bcde4736e90ff45a" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41577 } }
I20260620 21:54:32.017774 19483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:32.017818 19483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b375279556e4b90bcde4736e90ff45a, State: Initialized, Role: FOLLOWER
I20260620 21:54:32.018297 19483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b375279556e4b90bcde4736e90ff45a" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41577 } }
I20260620 21:54:32.018429 19483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:32.018528 19483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:32.018678 19483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:32.019872 19483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b375279556e4b90bcde4736e90ff45a" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41577 } }
I20260620 21:54:32.020269 19483 leader_election.cc:302] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b375279556e4b90bcde4736e90ff45a; no voters:
I20260620 21:54:32.020632 19483 leader_election.cc:288] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:32.020783 19488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:32.021075 19488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [term 1 LEADER]: Becoming Leader. State: Replica: 5b375279556e4b90bcde4736e90ff45a, State: Running, Role: LEADER
I20260620 21:54:32.021380 19488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b375279556e4b90bcde4736e90ff45a" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41577 } }
I20260620 21:54:32.021981 19483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:32.023272 19489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b375279556e4b90bcde4736e90ff45a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b375279556e4b90bcde4736e90ff45a" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41577 } } }
I20260620 21:54:32.023288 19490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b375279556e4b90bcde4736e90ff45a. Latest consensus state: current_term: 1 leader_uuid: "5b375279556e4b90bcde4736e90ff45a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b375279556e4b90bcde4736e90ff45a" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41577 } } }
I20260620 21:54:32.023488 19490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:32.023779 19489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:32.023895 19497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:32.027148 19497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:32.033349 19497 catalog_manager.cc:1347] Generated new cluster ID: da46c42d657244e9b0d6f18efc0f10c9
I20260620 21:54:32.033450 19497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:32.053174 19497 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:32.054507 19497 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:32.061311 19497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a: Generated new TSK 0
I20260620 21:54:32.062180 19497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:32.074041 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41577
--builtin_ntp_servers=127.0.193.212:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:54:32.197116 19507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:32.197467 19507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:32.197512 19507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:32.201982 19507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:32.202054 19507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:32.202114 19507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:54:32.207132 19507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:32.208292 19507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:32.209899 19507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:32.220393 19512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:32.220399 19513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:32.220396 19515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:32.220881 19507 server_base.cc:1034] running on GCE node
I20260620 21:54:32.221446 19507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:32.222182 19507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:32.223384 19507 hybrid_clock.cc:648] HybridClock initialized: now 1781992472223366 us; error 35 us; skew 500 ppm
I20260620 21:54:32.225896 19507 webserver.cc:460] Webserver started at http://127.0.193.193:40035/ using document root <none> and password file <none>
I20260620 21:54:32.226593 19507 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:32.226702 19507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:32.226997 19507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:32.229404 19507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "7375d8f8f71d40f582d917705e97f213"
format_stamp: "Formatted at 2026-06-20 21:54:32 on dist-test-slave-kk5c"
server_key: "1f777b1c032e45a45046bb5c67f53887"
server_key_iv: "a5936d9854d0867e3745e19c7b866f96"
server_key_version: "encryptionkey@0"
I20260620 21:54:32.230201 19507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "7375d8f8f71d40f582d917705e97f213"
format_stamp: "Formatted at 2026-06-20 21:54:32 on dist-test-slave-kk5c"
server_key: "1f777b1c032e45a45046bb5c67f53887"
server_key_iv: "a5936d9854d0867e3745e19c7b866f96"
server_key_version: "encryptionkey@0"
I20260620 21:54:32.234534 19507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 21:54:32.237675 19521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:32.239599 19507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:32.239776 19507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "7375d8f8f71d40f582d917705e97f213"
format_stamp: "Formatted at 2026-06-20 21:54:32 on dist-test-slave-kk5c"
server_key: "1f777b1c032e45a45046bb5c67f53887"
server_key_iv: "a5936d9854d0867e3745e19c7b866f96"
server_key_version: "encryptionkey@0"
I20260620 21:54:32.239926 19507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:32.254844 19507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:32.255599 19507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:32.255817 19507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:32.256413 19507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:32.257368 19507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:32.257440 19507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:32.257512 19507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:32.257565 19507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:32.267624 19507 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:37701
I20260620 21:54:32.267642 19634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:37701 every 8 connection(s)
I20260620 21:54:32.268637 19507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:54:32.271883 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 19507
I20260620 21:54:32.272035 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:54:32.272522 775 external_mini_cluster.cc:1428] Setting key 355d513629046f8e7a6c91764ddf12ad
I20260620 21:54:32.275151 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41577
--builtin_ntp_servers=127.0.193.212:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:54:32.280609 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.270698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:50613 (local address 127.0.193.254:41577)
0620 21:54:32.270873 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:32.270879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:32.272468 (+ 1589us) server_negotiation.cc:407] Connection header received
0620 21:54:32.274681 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:32.274685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:32.274736 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:32.274830 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:32.276029 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.276754 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.278075 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.278510 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.278646 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.279962 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:32.279968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:32.279972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:32.279992 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:32.280023 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:32.280037 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:32.280130 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:32.280291 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 21:54:32.280388 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 21:54:32.281471 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.271252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41577 (local address 127.0.193.193:50613)
0620 21:54:32.272294 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:32.272354 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:32.274398 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:32.275066 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:32.275074 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:32.275169 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:32.275864 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.275880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.276932 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.276934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.277916 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.277927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.278678 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.278681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.278840 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.279663 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:54:32.279665 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:32.279711 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:54:32.279786 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:32.280135 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:32.280141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:32.280145 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 21:54:32.280241 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 21:54:32.280492 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":133,"threads_started":1}
I20260620 21:54:32.282953 19635 heartbeater.cc:347] Connected to a master server at 127.0.193.254:41577
I20260620 21:54:32.283288 19635 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:32.284036 19635 heartbeater.cc:511] Master 127.0.193.254:41577 requested a full tablet report, sending...
I20260620 21:54:32.285949 19444 ts_manager.cc:194] Registered new tserver with Master: 7375d8f8f71d40f582d917705e97f213 (127.0.193.193:37701)
I20260620 21:54:32.287142 19444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.193.193:50613
W20260620 21:54:32.392431 19638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:32.392720 19638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:32.392761 19638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:32.397081 19638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:32.397157 19638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:32.397217 19638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:54:32.402726 19638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:32.404031 19638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:32.405504 19638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:32.416384 19643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:32.416384 19646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:32.416390 19644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:32.416996 19638 server_base.cc:1034] running on GCE node
I20260620 21:54:32.417455 19638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:32.418159 19638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:32.419356 19638 hybrid_clock.cc:648] HybridClock initialized: now 1781992472419334 us; error 34 us; skew 500 ppm
I20260620 21:54:32.421424 19638 webserver.cc:460] Webserver started at http://127.0.193.194:39521/ using document root <none> and password file <none>
I20260620 21:54:32.422142 19638 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:32.422233 19638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:32.422467 19638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:32.424849 19638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "e0eaa6871a414d9fac7893aa5deff072"
format_stamp: "Formatted at 2026-06-20 21:54:32 on dist-test-slave-kk5c"
server_key: "e50c68bf98a925848d660ec485e0a0a6"
server_key_iv: "b2b7330bb161ea12ad236edc0d916739"
server_key_version: "encryptionkey@0"
I20260620 21:54:32.425457 19638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "e0eaa6871a414d9fac7893aa5deff072"
format_stamp: "Formatted at 2026-06-20 21:54:32 on dist-test-slave-kk5c"
server_key: "e50c68bf98a925848d660ec485e0a0a6"
server_key_iv: "b2b7330bb161ea12ad236edc0d916739"
server_key_version: "encryptionkey@0"
I20260620 21:54:32.429713 19638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:54:32.432960 19652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:32.434561 19638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:54:32.434746 19638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "e0eaa6871a414d9fac7893aa5deff072"
format_stamp: "Formatted at 2026-06-20 21:54:32 on dist-test-slave-kk5c"
server_key: "e50c68bf98a925848d660ec485e0a0a6"
server_key_iv: "b2b7330bb161ea12ad236edc0d916739"
server_key_version: "encryptionkey@0"
I20260620 21:54:32.434890 19638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:32.453574 19638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:32.454319 19638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:32.454524 19638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:32.455133 19638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:32.456118 19638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:32.456189 19638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:32.456278 19638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:32.456321 19638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:32.465348 19638 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:38733
I20260620 21:54:32.465363 19765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:38733 every 8 connection(s)
I20260620 21:54:32.466287 19638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:54:32.472972 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 19638
I20260620 21:54:32.473136 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:54:32.473443 775 external_mini_cluster.cc:1428] Setting key cf264295b2830faea74c24eeafca8a8c
I20260620 21:54:32.475521 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41577
--builtin_ntp_servers=127.0.193.212:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:54:32.476702 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.468343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:36695 (local address 127.0.193.254:41577)
0620 21:54:32.468502 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:32.468506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:32.469547 (+ 1041us) server_negotiation.cc:407] Connection header received
0620 21:54:32.470939 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:32.470942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:32.470989 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:32.471068 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:32.472075 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.472726 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.474365 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.474613 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.474737 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.476148 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:32.476152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:32.476155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:32.476169 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:32.476198 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:32.476206 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:32.476292 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:32.476483 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 21:54:32.476553 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:54:32.477689 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.468728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41577 (local address 127.0.193.194:36695)
0620 21:54:32.469388 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:32.469434 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:32.470717 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:32.471326 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:32.471333 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:32.471416 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:32.471904 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.471917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.472957 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.472963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.474225 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.474236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.474778 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.474783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.474968 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.475860 (+ 892us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:54:32.475863 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:32.475907 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:54:32.475978 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:32.476306 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:32.476318 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:32.476325 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 21:54:32.476471 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 21:54:32.476789 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
I20260620 21:54:32.478945 19766 heartbeater.cc:347] Connected to a master server at 127.0.193.254:41577
I20260620 21:54:32.479240 19766 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:32.479940 19766 heartbeater.cc:511] Master 127.0.193.254:41577 requested a full tablet report, sending...
I20260620 21:54:32.481197 19444 ts_manager.cc:194] Registered new tserver with Master: e0eaa6871a414d9fac7893aa5deff072 (127.0.193.194:38733)
I20260620 21:54:32.481837 19444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.193.194:36695
W20260620 21:54:32.592178 19769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:32.592479 19769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:32.592520 19769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:32.596820 19769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:32.596899 19769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:32.596958 19769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:54:32.603406 19769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:32.604633 19769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:32.606333 19769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:32.617033 19774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:32.617036 19775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:32.617036 19777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:32.617298 19769 server_base.cc:1034] running on GCE node
I20260620 21:54:32.617929 19769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:32.618604 19769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:32.619781 19769 hybrid_clock.cc:648] HybridClock initialized: now 1781992472619725 us; error 64 us; skew 500 ppm
I20260620 21:54:32.622288 19769 webserver.cc:460] Webserver started at http://127.0.193.195:37497/ using document root <none> and password file <none>
I20260620 21:54:32.623148 19769 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:32.623287 19769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:32.623600 19769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:32.625862 19769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "2b251c6c8648435199c9dd16aa7a7cb0"
format_stamp: "Formatted at 2026-06-20 21:54:32 on dist-test-slave-kk5c"
server_key: "d682670b6676cc5443e098cb6812cb08"
server_key_iv: "f17f277be969573065b587ea4a1ba035"
server_key_version: "encryptionkey@0"
I20260620 21:54:32.626482 19769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "2b251c6c8648435199c9dd16aa7a7cb0"
format_stamp: "Formatted at 2026-06-20 21:54:32 on dist-test-slave-kk5c"
server_key: "d682670b6676cc5443e098cb6812cb08"
server_key_iv: "f17f277be969573065b587ea4a1ba035"
server_key_version: "encryptionkey@0"
I20260620 21:54:32.630613 19769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:54:32.633755 19783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:32.635545 19769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 21:54:32.635717 19769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "2b251c6c8648435199c9dd16aa7a7cb0"
format_stamp: "Formatted at 2026-06-20 21:54:32 on dist-test-slave-kk5c"
server_key: "d682670b6676cc5443e098cb6812cb08"
server_key_iv: "f17f277be969573065b587ea4a1ba035"
server_key_version: "encryptionkey@0"
I20260620 21:54:32.635867 19769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:32.656350 19769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:32.657119 19769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:32.657337 19769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:32.657953 19769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:32.659073 19769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:32.659154 19769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:32.659250 19769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:32.659304 19769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:32.669674 19769 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:39269
I20260620 21:54:32.669709 19896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:39269 every 8 connection(s)
I20260620 21:54:32.670651 19769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:54:32.672920 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 19769
I20260620 21:54:32.673034 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:54:32.673333 775 external_mini_cluster.cc:1428] Setting key fca84d214c5ce67e69cab2e14238e122
I20260620 21:54:32.681238 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.672663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:60229 (local address 127.0.193.254:41577)
0620 21:54:32.672879 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:32.672884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:32.674413 (+ 1529us) server_negotiation.cc:407] Connection header received
0620 21:54:32.676157 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:32.676159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:32.676204 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:32.676274 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:32.677310 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.677889 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.679071 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.679412 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.679526 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.680691 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:32.680694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:32.680696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:32.680710 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:32.680730 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:32.680738 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:32.680806 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:32.681055 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 21:54:32.681121 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:54:32.682257 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.673369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41577 (local address 127.0.193.195:60229)
0620 21:54:32.674208 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:32.674263 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:32.675967 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:32.676517 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:32.676525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:32.676612 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:32.677169 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.677183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.678038 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.678042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.678942 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.678951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.679621 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.679625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.679798 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.680416 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:54:32.680418 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:32.680469 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:54:32.680537 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:32.680875 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:32.680884 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:32.680889 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:54:32.681011 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 21:54:32.681250 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":139,"threads_started":1}
I20260620 21:54:32.683704 19897 heartbeater.cc:347] Connected to a master server at 127.0.193.254:41577
I20260620 21:54:32.684015 19897 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:32.684628 19897 heartbeater.cc:511] Master 127.0.193.254:41577 requested a full tablet report, sending...
I20260620 21:54:32.685792 19444 ts_manager.cc:194] Registered new tserver with Master: 2b251c6c8648435199c9dd16aa7a7cb0 (127.0.193.195:39269)
I20260620 21:54:32.686684 19444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.193.195:60229
I20260620 21:54:32.687891 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:54:32.694579 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.689960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.0.193.254:41577)
0620 21:54:32.690163 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:32.690170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:32.690304 (+ 134us) server_negotiation.cc:407] Connection header received
0620 21:54:32.690410 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:32.690412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:32.690452 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:32.690529 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:32.690844 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.691526 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.692509 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.692724 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.692816 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.693910 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:32.693914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:32.693916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:32.693928 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:32.693947 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:32.693956 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:32.694072 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:32.694196 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 21:54:32.694271 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 21:54:32.698410 19444 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:54:32.701785 19444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:54:32.717069 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.710857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.0.193.195:39269)
0620 21:54:32.712421 (+ 1564us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:32.712429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:32.712450 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:54:32.712507 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:32.712511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:32.712660 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:32.712824 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:32.713121 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.714377 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.715770 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.716400 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.716528 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.716738 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:32.716798 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 21:54:32.716904 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1477,"thread_start_us":167,"threads_started":1}
I20260620 21:54:32.717082 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.710744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:39269 (local address 127.0.0.1:51794)
0620 21:54:32.711283 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:32.711333 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:32.711558 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:32.712813 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:32.712818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:32.712840 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:32.712997 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.713012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.714524 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.714527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.715649 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.715659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.716529 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.716533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.716636 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.716671 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 21:54:32.716774 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 21:54:32.716894 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":84,"thread_start_us":151,"threads_started":1}
I20260620 21:54:32.717646 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.710801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:37701 (local address 127.0.0.1:46800)
0620 21:54:32.712000 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:32.712015 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:32.712143 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:32.713039 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:32.713042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:32.713059 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:32.713147 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.713152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.714569 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.714572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.715784 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.715794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.717066 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.717070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.717327 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.717351 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:54:32.717429 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 21:54:32.717497 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"mutex_wait_us":78,"spinlock_wait_cycles":10880,"thread_start_us":114,"threads_started":1}
I20260620 21:54:32.717748 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.711476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.0.193.193:37701)
0620 21:54:32.711895 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:32.711908 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:32.712258 (+ 350us) server_negotiation.cc:407] Connection header received
0620 21:54:32.712357 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:32.712365 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:32.712713 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:32.712877 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:32.713272 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.714453 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.715927 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.716725 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.716836 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.717003 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:32.717491 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 21:54:32.717598 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":117,"threads_started":1}
I20260620 21:54:32.718699 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.711656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:38733 (local address 127.0.0.1:54630)
0620 21:54:32.712023 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:32.712038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:32.712133 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:32.713105 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:32.713108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:32.713120 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:32.713185 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.713193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.714270 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.714274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.716517 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.716527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.718308 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.718311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.718408 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.718431 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 21:54:32.718499 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 21:54:32.718568 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":34,"spinlock_wait_cycles":768,"thread_start_us":112,"threads_started":1}
I20260620 21:54:32.719272 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.712000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.0.193.194:38733)
0620 21:54:32.712548 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:32.712563 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:32.712596 (+ 33us) server_negotiation.cc:407] Connection header received
0620 21:54:32.712672 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:32.712677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:32.712846 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:32.712971 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:32.713309 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.714143 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.716669 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.718009 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.718111 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.718308 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:32.718972 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 21:54:32.719085 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":128,"threads_started":1}
I20260620 21:54:32.720541 19831 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4803b53ed0475c87157c366e396230 (DEFAULT_TABLE table=test-table [id=72686fe0071b44948fabcb447f520921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:32.720806 19569 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4803b53ed0475c87157c366e396230 (DEFAULT_TABLE table=test-table [id=72686fe0071b44948fabcb447f520921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:32.721710 19569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4803b53ed0475c87157c366e396230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:32.721711 19831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4803b53ed0475c87157c366e396230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:32.721815 19700 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4803b53ed0475c87157c366e396230 (DEFAULT_TABLE table=test-table [id=72686fe0071b44948fabcb447f520921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:32.722751 19700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4803b53ed0475c87157c366e396230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:32.728875 19914 tablet_bootstrap.cc:492] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213: Bootstrap starting.
I20260620 21:54:32.729569 19915 tablet_bootstrap.cc:492] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0: Bootstrap starting.
I20260620 21:54:32.730284 19916 tablet_bootstrap.cc:492] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072: Bootstrap starting.
I20260620 21:54:32.731539 19915 tablet_bootstrap.cc:654] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:32.731530 19914 tablet_bootstrap.cc:654] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:32.732010 19916 tablet_bootstrap.cc:654] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:32.732445 19915 log.cc:824] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:32.732438 19914 log.cc:824] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:32.732733 19916 log.cc:824] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:32.734719 19914 tablet_bootstrap.cc:492] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213: No bootstrap required, opened a new log
I20260620 21:54:32.734719 19915 tablet_bootstrap.cc:492] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0: No bootstrap required, opened a new log
I20260620 21:54:32.734913 19916 tablet_bootstrap.cc:492] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072: No bootstrap required, opened a new log
I20260620 21:54:32.734943 19914 ts_tablet_manager.cc:1397] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:54:32.734943 19915 ts_tablet_manager.cc:1397] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:54:32.735129 19916 ts_tablet_manager.cc:1397] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:54:32.738727 19916 raft_consensus.cc:348] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b251c6c8648435199c9dd16aa7a7cb0" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39269 } } peers { permanent_uuid: "7375d8f8f71d40f582d917705e97f213" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37701 } } peers { permanent_uuid: "e0eaa6871a414d9fac7893aa5deff072" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 38733 } }
I20260620 21:54:32.739020 19916 raft_consensus.cc:374] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:32.739076 19916 raft_consensus.cc:729] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0eaa6871a414d9fac7893aa5deff072, State: Initialized, Role: FOLLOWER
I20260620 21:54:32.739388 19915 raft_consensus.cc:348] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b251c6c8648435199c9dd16aa7a7cb0" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39269 } } peers { permanent_uuid: "7375d8f8f71d40f582d917705e97f213" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37701 } } peers { permanent_uuid: "e0eaa6871a414d9fac7893aa5deff072" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 38733 } }
I20260620 21:54:32.739388 19914 raft_consensus.cc:348] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b251c6c8648435199c9dd16aa7a7cb0" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39269 } } peers { permanent_uuid: "7375d8f8f71d40f582d917705e97f213" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37701 } } peers { permanent_uuid: "e0eaa6871a414d9fac7893aa5deff072" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 38733 } }
I20260620 21:54:32.739652 19914 raft_consensus.cc:374] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:32.739652 19915 raft_consensus.cc:374] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:32.739708 19915 raft_consensus.cc:729] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b251c6c8648435199c9dd16aa7a7cb0, State: Initialized, Role: FOLLOWER
I20260620 21:54:32.739708 19914 raft_consensus.cc:729] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7375d8f8f71d40f582d917705e97f213, State: Initialized, Role: FOLLOWER
I20260620 21:54:32.739851 19916 consensus_queue.cc:260] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b251c6c8648435199c9dd16aa7a7cb0" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39269 } } peers { permanent_uuid: "7375d8f8f71d40f582d917705e97f213" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37701 } } peers { permanent_uuid: "e0eaa6871a414d9fac7893aa5deff072" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 38733 } }
I20260620 21:54:32.740135 19914 consensus_queue.cc:260] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b251c6c8648435199c9dd16aa7a7cb0" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39269 } } peers { permanent_uuid: "7375d8f8f71d40f582d917705e97f213" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37701 } } peers { permanent_uuid: "e0eaa6871a414d9fac7893aa5deff072" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 38733 } }
I20260620 21:54:32.740211 19915 consensus_queue.cc:260] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b251c6c8648435199c9dd16aa7a7cb0" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39269 } } peers { permanent_uuid: "7375d8f8f71d40f582d917705e97f213" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37701 } } peers { permanent_uuid: "e0eaa6871a414d9fac7893aa5deff072" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 38733 } }
I20260620 21:54:32.741117 19914 ts_tablet_manager.cc:1428] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260620 21:54:32.741117 19915 ts_tablet_manager.cc:1428] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260620 21:54:32.741186 19897 heartbeater.cc:503] Master 127.0.193.254:41577 was elected leader, sending a full tablet report...
I20260620 21:54:32.741607 19916 ts_tablet_manager.cc:1428] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:54:32.741707 19766 heartbeater.cc:503] Master 127.0.193.254:41577 was elected leader, sending a full tablet report...
I20260620 21:54:32.742092 19635 heartbeater.cc:503] Master 127.0.193.254:41577 was elected leader, sending a full tablet report...
I20260620 21:54:32.743268 19921 raft_consensus.cc:482] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:54:32.743409 19921 raft_consensus.cc:504] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b251c6c8648435199c9dd16aa7a7cb0" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39269 } } peers { permanent_uuid: "7375d8f8f71d40f582d917705e97f213" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37701 } } peers { permanent_uuid: "e0eaa6871a414d9fac7893aa5deff072" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 38733 } }
I20260620 21:54:32.745687 19921 leader_election.cc:288] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b251c6c8648435199c9dd16aa7a7cb0 (127.0.193.195:39269), e0eaa6871a414d9fac7893aa5deff072 (127.0.193.194:38733)
I20260620 21:54:32.748970 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.745385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:60749 (local address 127.0.193.195:39269)
0620 21:54:32.745513 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:32.745518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:32.745530 (+ 12us) server_negotiation.cc:407] Connection header received
0620 21:54:32.745574 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:32.745577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:32.745623 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:32.745713 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:32.746111 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.746817 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.748019 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.748422 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.748556 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.748603 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:32.748784 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 21:54:32.748860 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 21:54:32.748970 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.744917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:39269 (local address 127.0.193.193:60749)
0620 21:54:32.745037 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:32.745054 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:32.745129 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:32.745831 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:32.745835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:32.745854 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:32.745977 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.745985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.746949 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.746952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.747893 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.747903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.748552 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.748556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.748653 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.748674 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 21:54:32.748770 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 21:54:32.748843 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":16}
I20260620 21:54:32.749581 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4803b53ed0475c87157c366e396230" candidate_uuid: "7375d8f8f71d40f582d917705e97f213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b251c6c8648435199c9dd16aa7a7cb0" is_pre_election: true
I20260620 21:54:32.749882 19851 raft_consensus.cc:2454] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7375d8f8f71d40f582d917705e97f213 in term 0.
I20260620 21:54:32.750061 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.745755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:47443 (local address 127.0.193.194:38733)
0620 21:54:32.745879 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:32.745884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:32.746657 (+ 773us) server_negotiation.cc:407] Connection header received
0620 21:54:32.746829 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:32.746832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:32.746883 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:32.746949 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:32.747357 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.747852 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.748865 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.749425 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.749571 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.749614 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:32.749873 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 21:54:32.749956 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:54:32.750058 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.745381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:38733 (local address 127.0.193.193:47443)
0620 21:54:32.746573 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:32.746589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:32.746715 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:32.747000 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:32.747002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:32.747020 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:32.747237 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.747243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.747969 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.747974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.748741 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:32.748749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.749574 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.749578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:32.749684 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.749714 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 21:54:32.749846 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 21:54:32.749917 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"mutex_wait_us":43,"thread_start_us":121,"threads_started":1}
I20260620 21:54:32.750653 19523 leader_election.cc:302] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b251c6c8648435199c9dd16aa7a7cb0, 7375d8f8f71d40f582d917705e97f213; no voters:
I20260620 21:54:32.750730 19720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4803b53ed0475c87157c366e396230" candidate_uuid: "7375d8f8f71d40f582d917705e97f213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0eaa6871a414d9fac7893aa5deff072" is_pre_election: true
I20260620 21:54:32.750974 19921 raft_consensus.cc:2781] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:54:32.751076 19921 raft_consensus.cc:482] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:54:32.751060 19720 raft_consensus.cc:2454] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7375d8f8f71d40f582d917705e97f213 in term 0.
I20260620 21:54:32.751134 19921 raft_consensus.cc:3037] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:32.752254 19921 raft_consensus.cc:504] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b251c6c8648435199c9dd16aa7a7cb0" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39269 } } peers { permanent_uuid: "7375d8f8f71d40f582d917705e97f213" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37701 } } peers { permanent_uuid: "e0eaa6871a414d9fac7893aa5deff072" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 38733 } }
I20260620 21:54:32.752622 19921 leader_election.cc:288] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [CANDIDATE]: Term 1 election: Requested vote from peers 2b251c6c8648435199c9dd16aa7a7cb0 (127.0.193.195:39269), e0eaa6871a414d9fac7893aa5deff072 (127.0.193.194:38733)
I20260620 21:54:32.753023 19851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4803b53ed0475c87157c366e396230" candidate_uuid: "7375d8f8f71d40f582d917705e97f213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b251c6c8648435199c9dd16aa7a7cb0"
I20260620 21:54:32.753160 19851 raft_consensus.cc:3037] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:32.754318 19851 raft_consensus.cc:2454] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7375d8f8f71d40f582d917705e97f213 in term 1.
I20260620 21:54:32.754626 19720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4803b53ed0475c87157c366e396230" candidate_uuid: "7375d8f8f71d40f582d917705e97f213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0eaa6871a414d9fac7893aa5deff072"
I20260620 21:54:32.754733 19523 leader_election.cc:302] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b251c6c8648435199c9dd16aa7a7cb0, 7375d8f8f71d40f582d917705e97f213; no voters:
I20260620 21:54:32.754778 19720 raft_consensus.cc:3037] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:32.755014 19921 raft_consensus.cc:2781] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:32.755331 19921 raft_consensus.cc:686] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [term 1 LEADER]: Becoming Leader. State: Replica: 7375d8f8f71d40f582d917705e97f213, State: Running, Role: LEADER
I20260620 21:54:32.755719 19921 consensus_queue.cc:237] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b251c6c8648435199c9dd16aa7a7cb0" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39269 } } peers { permanent_uuid: "7375d8f8f71d40f582d917705e97f213" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37701 } } peers { permanent_uuid: "e0eaa6871a414d9fac7893aa5deff072" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 38733 } }
I20260620 21:54:32.756170 19720 raft_consensus.cc:2454] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7375d8f8f71d40f582d917705e97f213 in term 1.
I20260620 21:54:32.759480 19446 catalog_manager.cc:5495] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7375d8f8f71d40f582d917705e97f213 (127.0.193.193). New cstate: current_term: 1 leader_uuid: "7375d8f8f71d40f582d917705e97f213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b251c6c8648435199c9dd16aa7a7cb0" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7375d8f8f71d40f582d917705e97f213" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 37701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0eaa6871a414d9fac7893aa5deff072" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 38733 } health_report { overall_health: UNKNOWN } } }
W20260620 21:54:32.769897 19636 tablet.cc:2369] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:54:32.784025 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.780333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.0.193.193:37701)
0620 21:54:32.780477 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:32.780484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:32.780501 (+ 17us) server_negotiation.cc:407] Connection header received
0620 21:54:32.780586 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:32.780588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:32.780644 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:32.780765 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:54:32.781032 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.781540 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.782439 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:32.782663 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:32.782778 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:32.783116 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:54:32.783563 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:54:32.783730 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:32.783824 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 21:54:32.783894 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:54:32.790617 19720 raft_consensus.cc:1261] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072 [term 1 FOLLOWER]: Refusing update from remote peer 7375d8f8f71d40f582d917705e97f213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:54:32.790617 19851 raft_consensus.cc:1261] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0 [term 1 FOLLOWER]: Refusing update from remote peer 7375d8f8f71d40f582d917705e97f213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:54:32.791543 19924 consensus_queue.cc:1035] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0eaa6871a414d9fac7893aa5deff072" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 38733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:54:32.791818 19921 consensus_queue.cc:1035] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b251c6c8648435199c9dd16aa7a7cb0" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 39269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:54:32.800810 19928 mvcc.cc:205] Tried to move back new op lower bound from 7299041168542150656 to 7299041168411017216. Current Snapshot: MvccSnapshot[applied={T|T < 7299041168542150656}]
I20260620 21:54:32.801004 19929 mvcc.cc:205] Tried to move back new op lower bound from 7299041168542150656 to 7299041168411017216. Current Snapshot: MvccSnapshot[applied={T|T < 7299041168542150656}]
I20260620 21:54:32.810290 19444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:54:32.810525 19444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:54:32.814013 19444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b375279556e4b90bcde4736e90ff45a: Sending DeleteTablet for 3 replicas of tablet 7e4803b53ed0475c87157c366e396230
I20260620 21:54:32.815101 19569 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4803b53ed0475c87157c366e396230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:32 UTC) from {username='slave'} at 127.0.0.1:46800
I20260620 21:54:32.815282 19831 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4803b53ed0475c87157c366e396230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:32 UTC) from {username='slave'} at 127.0.0.1:51794
I20260620 21:54:32.815281 19700 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4803b53ed0475c87157c366e396230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:32 UTC) from {username='slave'} at 127.0.0.1:54630
I20260620 21:54:32.815708 19937 tablet_replica.cc:330] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213: stopping tablet replica
I20260620 21:54:32.816072 19937 raft_consensus.cc:2229] T 7e4803b53ed0475c87157c366e396230 P 7375d8f8f71d40f582d917705e97f213 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:54:32.816216 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 19507
I20260620 21:54:32.819375 19939 tablet_replica.cc:330] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0: stopping tablet replica
I20260620 21:54:32.819464 19938 tablet_replica.cc:330] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072: stopping tablet replica
I20260620 21:54:32.819766 19939 raft_consensus.cc:2229] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:54:32.819818 19938 raft_consensus.cc:2229] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:54:32.820142 19939 raft_consensus.cc:2258] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:54:32.820148 19938 raft_consensus.cc:2258] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:54:32.821650 19938 ts_tablet_manager.cc:1905] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:54:32.824288 19938 ts_tablet_manager.cc:1918] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:54:32.824404 19938 log.cc:1196] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/7e4803b53ed0475c87157c366e396230
W20260620 21:54:32.824460 19433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:37701 (error 108)
I20260620 21:54:32.824868 19938 ts_tablet_manager.cc:1939] T 7e4803b53ed0475c87157c366e396230 P e0eaa6871a414d9fac7893aa5deff072: Deleting consensus metadata
I20260620 21:54:32.825408 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 19638
I20260620 21:54:32.825702 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.825322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:37701 (local address 127.0.0.1:46826)
0620 21:54:32.825534 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:32.825609 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:37701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260620 21:54:32.826428 19433 catalog_manager.cc:4584] TS 7375d8f8f71d40f582d917705e97f213 (127.0.193.193:37701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e4803b53ed0475c87157c366e396230: Network error: Client connection negotiation failed: client connection to 127.0.193.193:37701: connect: Connection refused (error 111)
I20260620 21:54:32.826485 19939 ts_tablet_manager.cc:1905] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:54:32.830600 19939 ts_tablet_manager.cc:1918] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:54:32.830961 19939 log.cc:1196] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/7e4803b53ed0475c87157c366e396230
I20260620 21:54:32.831394 19939 ts_tablet_manager.cc:1939] T 7e4803b53ed0475c87157c366e396230 P 2b251c6c8648435199c9dd16aa7a7cb0: Deleting consensus metadata
I20260620 21:54:32.832787 19432 catalog_manager.cc:4841] TS 2b251c6c8648435199c9dd16aa7a7cb0 (127.0.193.195:39269): tablet 7e4803b53ed0475c87157c366e396230 (table test-table [id=72686fe0071b44948fabcb447f520921]) successfully deleted
I20260620 21:54:32.832832 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 19769
I20260620 21:54:32.833647 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:32.833165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:38733 (local address 127.0.0.1:54632)
0620 21:54:32.833471 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:32.833538 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.194:38733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260620 21:54:32.838789 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 19416
2026-06-20T21:54:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1076 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:32 dist-test-slave-kk5c krb5kdc[19943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:32 dist-test-slave-kk5c krb5kdc[19943](info): set up 2 sockets
Jun 20 21:54:32 dist-test-slave-kk5c krb5kdc[19943](info): commencing operation
krb5kdc: starting...
W20260620 21:54:33.903395 775 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:33 dist-test-slave-kk5c krb5kdc[19943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:54:34.031044 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:45297
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:45297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:34.144120 19956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:34.144470 19956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:34.144549 19956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:34.148728 19956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:34.148846 19956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:34.148885 19956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:34.148917 19956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:34.148974 19956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:34.154906 19956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:45297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:34.156104 19956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:34.157214 19956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:34.165939 19964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:34.165932 19962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:34.165932 19961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:34.166185 19956 server_base.cc:1034] running on GCE node
I20260620 21:54:34.167054 19956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:34.168197 19956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:34.169368 19956 hybrid_clock.cc:648] HybridClock initialized: now 1781992474169335 us; error 51 us; skew 500 ppm
Jun 20 21:54:34 dist-test-slave-kk5c krb5kdc[19943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:34.172358 19956 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:54:34.173413 19956 webserver.cc:460] Webserver started at http://127.0.193.254:38777/ using document root <none> and password file <none>
I20260620 21:54:34.173969 19956 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:34.174031 19956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:34.174199 19956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:34.176417 19956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "34bd81b6c6b64d9c9539cdbf20c4cddd"
format_stamp: "Formatted at 2026-06-20 21:54:34 on dist-test-slave-kk5c"
server_key: "aa6e698d4fc3b6039a9c6bad8a895f36"
server_key_iv: "e23a0e4454f032d8e8402e1e9f2a40b7"
server_key_version: "encryptionkey@0"
I20260620 21:54:34.176977 19956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "34bd81b6c6b64d9c9539cdbf20c4cddd"
format_stamp: "Formatted at 2026-06-20 21:54:34 on dist-test-slave-kk5c"
server_key: "aa6e698d4fc3b6039a9c6bad8a895f36"
server_key_iv: "e23a0e4454f032d8e8402e1e9f2a40b7"
server_key_version: "encryptionkey@0"
I20260620 21:54:34.181463 19956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:54:34.184032 19971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:34.185312 19956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:54:34.185425 19956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "34bd81b6c6b64d9c9539cdbf20c4cddd"
format_stamp: "Formatted at 2026-06-20 21:54:34 on dist-test-slave-kk5c"
server_key: "aa6e698d4fc3b6039a9c6bad8a895f36"
server_key_iv: "e23a0e4454f032d8e8402e1e9f2a40b7"
server_key_version: "encryptionkey@0"
I20260620 21:54:34.185568 19956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:34.210681 19956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:34.214357 19956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:34.214591 19956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:34.222229 19956 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:45297
I20260620 21:54:34.222255 20023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:45297 every 8 connection(s)
I20260620 21:54:34.223270 19956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:34.226096 20024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:34.228262 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 19956
I20260620 21:54:34.228423 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:34.228731 775 external_mini_cluster.cc:1428] Setting key 804443a765e99c29b0b64187a0a3751c
I20260620 21:54:34.233285 20024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd: Bootstrap starting.
I20260620 21:54:34.235844 20024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:34.236701 20024 log.cc:824] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:54:34 dist-test-slave-kk5c krb5kdc[19943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:54:34.239547 20024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd: No bootstrap required, opened a new log
I20260620 21:54:34.244946 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:34.230305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.0.193.254:45297)
0620 21:54:34.230780 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:34.230795 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:34.230873 (+ 78us) server_negotiation.cc:407] Connection header received
0620 21:54:34.231619 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:34.231638 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:34.231985 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:34.232329 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:34.233165 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:34.234203 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:34.235095 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:34.235656 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:34.235837 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:34.239099 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:34.239105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:34.239119 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:34.239147 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:34.241366 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:34.241908 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:34.241917 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:34.241927 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:34.242092 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:34.242384 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:34.242388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:34.242392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:34.242976 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:34.243158 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:34.243569 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 21:54:34.243903 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":145,"threads_started":1}
I20260620 21:54:34.244750 20024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bd81b6c6b64d9c9539cdbf20c4cddd" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45297 } }
I20260620 21:54:34.245050 20024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:34.245095 20024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34bd81b6c6b64d9c9539cdbf20c4cddd, State: Initialized, Role: FOLLOWER
I20260620 21:54:34.245582 20024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bd81b6c6b64d9c9539cdbf20c4cddd" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45297 } }
I20260620 21:54:34.245741 20024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:34.245817 20024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:34.245923 20024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:34.246934 20024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bd81b6c6b64d9c9539cdbf20c4cddd" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45297 } }
I20260620 21:54:34.247381 20024 leader_election.cc:302] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34bd81b6c6b64d9c9539cdbf20c4cddd; no voters:
I20260620 21:54:34.247756 20024 leader_election.cc:288] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:34.247963 20029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:34.248276 20029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [term 1 LEADER]: Becoming Leader. State: Replica: 34bd81b6c6b64d9c9539cdbf20c4cddd, State: Running, Role: LEADER
I20260620 21:54:34.248642 20029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bd81b6c6b64d9c9539cdbf20c4cddd" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45297 } }
I20260620 21:54:34.249091 20024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:34.250329 20031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34bd81b6c6b64d9c9539cdbf20c4cddd. Latest consensus state: current_term: 1 leader_uuid: "34bd81b6c6b64d9c9539cdbf20c4cddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bd81b6c6b64d9c9539cdbf20c4cddd" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45297 } } }
I20260620 21:54:34.250460 20031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:34.250797 20030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34bd81b6c6b64d9c9539cdbf20c4cddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34bd81b6c6b64d9c9539cdbf20c4cddd" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45297 } } }
I20260620 21:54:34.250905 20030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:34.250851 20038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:34.254037 20038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:34.260823 20038 catalog_manager.cc:1347] Generated new cluster ID: 3d2f39ed67ae4b0f8aae8ffa9ce7d234
I20260620 21:54:34.260972 20038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:34.291582 20038 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:34.292719 20038 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:34.300252 20038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd: Generated new TSK 0
I20260620 21:54:34.301084 20038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
Entry for principal kudu/127.0.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab.
I20260620 21:54:34.339165 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:45297
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:34.475950 20050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:34.476262 20050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:34.476307 20050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:34.480587 20050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:34.480664 20050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:34.480764 20050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:54:34.486397 20050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.193.keytab
--principal=kudu/127.0.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:34.487594 20050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:34.488652 20050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:34.498929 20055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:34.498912 20058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:34.499058 20050 server_base.cc:1034] running on GCE node
W20260620 21:54:34.498912 20056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:34.499622 20050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:34.500391 20050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:34.501626 20050 hybrid_clock.cc:648] HybridClock initialized: now 1781992474501607 us; error 33 us; skew 500 ppm
Jun 20 21:54:34 dist-test-slave-kk5c krb5kdc[19943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:34.504846 20050 init.cc:377] Logged in from keytab as kudu/127.0.193.193@KRBTEST.COM (short username kudu)
I20260620 21:54:34.505918 20050 webserver.cc:460] Webserver started at http://127.0.193.193:37121/ using document root <none> and password file <none>
I20260620 21:54:34.506558 20050 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:34.506675 20050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:34.506922 20050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:34.509166 20050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed"
format_stamp: "Formatted at 2026-06-20 21:54:34 on dist-test-slave-kk5c"
server_key: "e9ac32bca675d2dc4f52b8e508e09d4c"
server_key_iv: "6044840b7c53d99b1dcda88dfc5bc8b5"
server_key_version: "encryptionkey@0"
I20260620 21:54:34.509773 20050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed"
format_stamp: "Formatted at 2026-06-20 21:54:34 on dist-test-slave-kk5c"
server_key: "e9ac32bca675d2dc4f52b8e508e09d4c"
server_key_iv: "6044840b7c53d99b1dcda88dfc5bc8b5"
server_key_version: "encryptionkey@0"
I20260620 21:54:34.513819 20050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:54:34.516745 20065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:34.518162 20050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:54:34.518334 20050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed"
format_stamp: "Formatted at 2026-06-20 21:54:34 on dist-test-slave-kk5c"
server_key: "e9ac32bca675d2dc4f52b8e508e09d4c"
server_key_iv: "6044840b7c53d99b1dcda88dfc5bc8b5"
server_key_version: "encryptionkey@0"
I20260620 21:54:34.518472 20050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:34.537741 20050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:34.541226 20050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:34.541464 20050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:34.542117 20050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:34.543018 20050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:34.543092 20050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:34.543166 20050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:34.543246 20050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:34.553350 20050 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:46715
I20260620 21:54:34.553372 20178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:46715 every 8 connection(s)
I20260620 21:54:34.554306 20050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:54:34.557808 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 20050
I20260620 21:54:34.557963 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:54:34.558254 775 external_mini_cluster.cc:1428] Setting key c38618968c5ff8f6657892cf22cab766
Jun 20 21:54:34 dist-test-slave-kk5c krb5kdc[19943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.193@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.194@KRBTEST.COM; defaulting to no policy
I20260620 21:54:34.570408 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:34.556280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:47013 (local address 127.0.193.254:45297)
0620 21:54:34.556522 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:34.556526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:34.557566 (+ 1040us) server_negotiation.cc:407] Connection header received
0620 21:54:34.559319 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:34.559322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:34.559376 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:34.559456 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:34.560872 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:34.561627 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:34.562929 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:34.563384 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:34.563517 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:34.566116 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:34.566122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:34.566125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:34.566160 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:34.567937 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:34.568625 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:34.568628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:34.568629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:34.568710 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:34.569114 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:34.569117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:34.569118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:34.569307 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:34.569421 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:34.570153 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 21:54:34.570278 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.194@KRBTEST.COM" created.
I20260620 21:54:34.571456 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:34.556681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:45297 (local address 127.0.193.193:47013)
0620 21:54:34.557192 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:34.557238 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:34.559006 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:34.559699 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:34.559710 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:34.560208 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:34.560717 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:34.560731 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:34.561801 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:34.561812 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:34.562754 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:34.562764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:34.563577 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:34.563580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:34.563741 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:34.564445 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:54:34.564462 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:34.565927 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:34.568113 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:34.568128 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:34.568142 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:34.568430 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:34.568810 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:34.568813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:34.568814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:34.568978 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:34.569409 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:34.569422 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:34.569756 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 21:54:34.570021 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 21:54:34.570332 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":123,"threads_started":1}
I20260620 21:54:34.572835 20179 heartbeater.cc:347] Connected to a master server at 127.0.193.254:45297
I20260620 21:54:34.573086 20179 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:34.573594 20179 heartbeater.cc:511] Master 127.0.193.254:45297 requested a full tablet report, sending...
I20260620 21:54:34.575376 19988 ts_manager.cc:194] Registered new tserver with Master: 3a08b4775e2a494a8ebdb57bbe4bb2ed (127.0.193.193:46715)
I20260620 21:54:34.577165 19988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.193@KRBTEST.COM'} at 127.0.193.193:47013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
Entry for principal kudu/127.0.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab.
I20260620 21:54:34.590302 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:45297
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:34.708731 20184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:34.709059 20184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:34.709105 20184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:34.713385 20184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:34.713464 20184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:34.713528 20184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:54:34.719348 20184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.194.keytab
--principal=kudu/127.0.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:34.720415 20184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:34.721431 20184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:34.731884 20189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:34.732028 20190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:34.731859 20192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:34.732143 20184 server_base.cc:1034] running on GCE node
I20260620 21:54:34.732620 20184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:34.733566 20184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:34.734933 20184 hybrid_clock.cc:648] HybridClock initialized: now 1781992474734888 us; error 53 us; skew 500 ppm
Jun 20 21:54:34 dist-test-slave-kk5c krb5kdc[19943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:34.738168 20184 init.cc:377] Logged in from keytab as kudu/127.0.193.194@KRBTEST.COM (short username kudu)
I20260620 21:54:34.739259 20184 webserver.cc:460] Webserver started at http://127.0.193.194:37187/ using document root <none> and password file <none>
I20260620 21:54:34.739861 20184 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:34.739976 20184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:34.740216 20184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:34.742488 20184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "81f49bc09d114d95931d7a89a8b602e6"
format_stamp: "Formatted at 2026-06-20 21:54:34 on dist-test-slave-kk5c"
server_key: "69c5622580e012678534db8cd80fa329"
server_key_iv: "553c0d803873acb757e094e05db1b190"
server_key_version: "encryptionkey@0"
I20260620 21:54:34.743098 20184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "81f49bc09d114d95931d7a89a8b602e6"
format_stamp: "Formatted at 2026-06-20 21:54:34 on dist-test-slave-kk5c"
server_key: "69c5622580e012678534db8cd80fa329"
server_key_iv: "553c0d803873acb757e094e05db1b190"
server_key_version: "encryptionkey@0"
I20260620 21:54:34.747170 20184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:54:34.750301 20199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:34.751768 20184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 21:54:34.751916 20184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "81f49bc09d114d95931d7a89a8b602e6"
format_stamp: "Formatted at 2026-06-20 21:54:34 on dist-test-slave-kk5c"
server_key: "69c5622580e012678534db8cd80fa329"
server_key_iv: "553c0d803873acb757e094e05db1b190"
server_key_version: "encryptionkey@0"
I20260620 21:54:34.752054 20184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:34.775158 20184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:34.778836 20184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:34.779070 20184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:34.779765 20184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:34.780700 20184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:34.780778 20184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:34.780849 20184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:34.780897 20184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:34.791651 20184 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:36443
I20260620 21:54:34.791668 20312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:36443 every 8 connection(s)
I20260620 21:54:34.792611 20184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:54:34.797684 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 20184
I20260620 21:54:34.797816 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:54:34.798101 775 external_mini_cluster.cc:1428] Setting key 43ef480faaca384daf1ef1a6f2258903
Jun 20 21:54:34 dist-test-slave-kk5c krb5kdc[19943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.194@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.195@KRBTEST.COM; defaulting to no policy
I20260620 21:54:34.809178 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:34.794472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:36913 (local address 127.0.193.254:45297)
0620 21:54:34.794651 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:34.794656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:34.795606 (+ 950us) server_negotiation.cc:407] Connection header received
0620 21:54:34.796960 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:34.796963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:34.797014 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:34.797095 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:34.798762 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:34.799527 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:34.801339 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:34.801566 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:34.801686 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:34.804825 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:34.804828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:34.804830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:34.804860 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:34.806528 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:34.807294 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:34.807297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:34.807299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:34.807386 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:34.807831 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:34.807834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:34.807835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:34.808092 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:34.808178 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:34.808917 (+ 739us) server_negotiation.cc:299] Negotiation successful
0620 21:54:34.809047 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 21:54:34.810119 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:34.794905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:45297 (local address 127.0.193.194:36913)
0620 21:54:34.795454 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:34.795498 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:34.796684 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:34.797260 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:34.797267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:34.797891 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:34.798587 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:34.798604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:34.799671 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:34.799675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:34.801204 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:34.801219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:34.801767 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:34.801775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:34.801950 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:34.802607 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:54:34.802623 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:34.804583 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:34.806674 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:34.806686 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:34.806707 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:34.807115 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:34.807500 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:34.807505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:34.807509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:34.807729 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:34.808210 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:34.808218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:34.808604 (+ 386us) client_negotiation.cc:769] Sending connection context
0620 21:54:34.808843 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 21:54:34.809138 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.195@KRBTEST.COM" created.
I20260620 21:54:34.811781 20313 heartbeater.cc:347] Connected to a master server at 127.0.193.254:45297
I20260620 21:54:34.812073 20313 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:34.812729 20313 heartbeater.cc:511] Master 127.0.193.254:45297 requested a full tablet report, sending...
I20260620 21:54:34.814123 19988 ts_manager.cc:194] Registered new tserver with Master: 81f49bc09d114d95931d7a89a8b602e6 (127.0.193.194:36443)
I20260620 21:54:34.814814 19988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.194@KRBTEST.COM'} at 127.0.193.194:36913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
Entry for principal kudu/127.0.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab.
I20260620 21:54:34.829950 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:45297
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:34.950174 20318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:34.950480 20318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:34.950527 20318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:34.955231 20318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:34.955336 20318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:34.955435 20318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:54:34.961458 20318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/krb5kdc/kudu_127.0.193.195.keytab
--principal=kudu/127.0.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:34.962673 20318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:34.963833 20318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:34.974910 20323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:34.974953 20324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:34.974956 20326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:34.975438 20318 server_base.cc:1034] running on GCE node
I20260620 21:54:34.975934 20318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:34.976694 20318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:34.977919 20318 hybrid_clock.cc:648] HybridClock initialized: now 1781992474977883 us; error 53 us; skew 500 ppm
Jun 20 21:54:34 dist-test-slave-kk5c krb5kdc[19943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:34.981096 20318 init.cc:377] Logged in from keytab as kudu/127.0.193.195@KRBTEST.COM (short username kudu)
I20260620 21:54:34.982132 20318 webserver.cc:460] Webserver started at http://127.0.193.195:43931/ using document root <none> and password file <none>
I20260620 21:54:34.982700 20318 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:34.982782 20318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:34.982960 20318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:34.985395 20318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b"
format_stamp: "Formatted at 2026-06-20 21:54:34 on dist-test-slave-kk5c"
server_key: "ce45daf825ff4ab5d57c60a657be74bc"
server_key_iv: "eb60de76771bdb6a299047dab7a2bfc9"
server_key_version: "encryptionkey@0"
I20260620 21:54:34.985975 20318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b"
format_stamp: "Formatted at 2026-06-20 21:54:34 on dist-test-slave-kk5c"
server_key: "ce45daf825ff4ab5d57c60a657be74bc"
server_key_iv: "eb60de76771bdb6a299047dab7a2bfc9"
server_key_version: "encryptionkey@0"
I20260620 21:54:34.990520 20318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 21:54:34.993285 20333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:34.994683 20318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:54:34.994836 20318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b"
format_stamp: "Formatted at 2026-06-20 21:54:34 on dist-test-slave-kk5c"
server_key: "ce45daf825ff4ab5d57c60a657be74bc"
server_key_iv: "eb60de76771bdb6a299047dab7a2bfc9"
server_key_version: "encryptionkey@0"
I20260620 21:54:34.994964 20318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:35.016114 20318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:35.019709 20318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:35.019929 20318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:35.020561 20318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:35.021426 20318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:35.021473 20318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:35.021565 20318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:35.021606 20318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:35.030982 20318 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:37503
I20260620 21:54:35.031000 20446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:37503 every 8 connection(s)
I20260620 21:54:35.032012 20318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:54:35.037675 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 20318
I20260620 21:54:35.037847 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:54:35.038148 775 external_mini_cluster.cc:1428] Setting key e46ff0d20fd5609fff564a8c7d945e96
Jun 20 21:54:35 dist-test-slave-kk5c krb5kdc[19943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.195@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:54:35.047322 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.034155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:56409 (local address 127.0.193.254:45297)
0620 21:54:35.034315 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:35.034319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:35.035232 (+ 913us) server_negotiation.cc:407] Connection header received
0620 21:54:35.036639 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:35.036642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:35.036704 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:35.036814 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:35.038126 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.038831 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.040056 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.040370 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.040499 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.043150 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:35.043153 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:35.043155 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:35.043181 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:35.044807 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:35.045536 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:35.045540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:35.045541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:35.045631 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:35.046058 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:35.046061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:35.046062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:35.046261 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:35.046355 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:35.046949 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 21:54:35.047071 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 21:54:35.048328 20449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.034543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:45297 (local address 127.0.193.195:56409)
0620 21:54:35.035032 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:35.035075 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:35.036410 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:35.037003 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:35.037011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:35.037395 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:35.037970 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:35.037988 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.038969 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.038972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:35.039923 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:35.039933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.040541 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.040545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:35.040735 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.041321 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:54:35.041338 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:35.042972 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:35.045088 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:35.045093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:35.045108 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:35.045391 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:35.045779 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:54:35.045782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:54:35.045784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:54:35.045955 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:54:35.046398 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:35.046405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:35.046690 (+ 285us) client_negotiation.cc:769] Sending connection context
0620 21:54:35.046879 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 21:54:35.047161 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":128,"threads_started":1}
I20260620 21:54:35.049705 20447 heartbeater.cc:347] Connected to a master server at 127.0.193.254:45297
I20260620 21:54:35.050050 20447 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:35.050660 20447 heartbeater.cc:511] Master 127.0.193.254:45297 requested a full tablet report, sending...
I20260620 21:54:35.051941 19988 ts_manager.cc:194] Registered new tserver with Master: d9bcd15b80dc4cff80fd0f98be6a2a6b (127.0.193.195:37503)
I20260620 21:54:35.052333 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:54:35.052623 19988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.193.195@KRBTEST.COM'} at 127.0.193.195:56409
I20260620 21:54:35.062055 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.054475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.0.193.254:45297)
0620 21:54:35.054616 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:35.054620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:35.054752 (+ 132us) server_negotiation.cc:407] Connection header received
0620 21:54:35.054831 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:35.054833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:35.054870 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:35.054935 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:35.055733 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.056271 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.057189 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.057418 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.057502 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.058910 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:35.058913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:35.058915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:35.058939 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:35.060476 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:35.060905 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:35.060907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:35.060909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:35.060990 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:35.061248 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:35.061251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:35.061252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:35.061495 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:35.061608 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:35.061832 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 21:54:35.061935 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 21:54:35.065199 19988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:54:35.068315 19988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:54:35.084200 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.077463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:46715 (local address 127.0.0.1:51732)
0620 21:54:35.078066 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:35.078081 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:35.078173 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:35.079137 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:35.079143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:35.079167 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:35.079373 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:35.079381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.080929 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.080935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:35.082448 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:35.082459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.083553 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.083557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:35.083663 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.083708 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 21:54:35.083876 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 21:54:35.084018 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":33,"thread_start_us":82,"threads_started":1}
I20260620 21:54:35.084182 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.078029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.0.193.193:46715)
0620 21:54:35.078626 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:35.078633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:35.078659 (+ 26us) server_negotiation.cc:407] Connection header received
0620 21:54:35.078740 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:35.078745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:35.078965 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:35.079122 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:35.079517 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.080775 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.082584 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.083428 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.083552 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.083714 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:35.083913 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 21:54:35.084014 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":98,"threads_started":1}
I20260620 21:54:35.084450 20459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.077332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.0.193.194:36443)
0620 21:54:35.078868 (+ 1536us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:35.078877 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:35.078895 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:54:35.078954 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:35.078958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:35.079138 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:35.079362 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:35.079584 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.080739 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.082779 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.083524 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.083998 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.084150 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:35.084198 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 21:54:35.084299 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1447,"thread_start_us":154,"threads_started":1}
I20260620 21:54:35.084712 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.077974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:37503 (local address 127.0.0.1:55010)
0620 21:54:35.078413 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:35.078426 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:35.078588 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:35.079317 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:35.079322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:35.079340 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:35.079480 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:35.079484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.082079 (+ 2595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.082085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:35.083082 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:35.083092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.084310 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.084313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:35.084397 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.084417 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 21:54:35.084493 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 21:54:35.084565 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":219,"spinlock_wait_cycles":1920,"thread_start_us":114,"threads_started":1}
I20260620 21:54:35.085181 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.078396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55010 (local address 127.0.193.195:37503)
0620 21:54:35.078744 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:35.078757 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:35.078776 (+ 19us) server_negotiation.cc:407] Connection header received
0620 21:54:35.078837 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:35.078842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:35.078978 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:35.079122 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:35.080913 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.081941 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.083246 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.084029 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.084126 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.084238 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:35.084943 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 21:54:35.085047 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":146,"threads_started":1}
I20260620 21:54:35.085188 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.077176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:36443 (local address 127.0.0.1:39338)
0620 21:54:35.077560 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:35.077609 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:35.077748 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:35.079333 (+ 1585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:35.079336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:35.079347 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:35.079475 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:35.079483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.081513 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.081516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:35.082668 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:35.082675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.083662 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.083666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:35.083768 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.083808 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 21:54:35.083861 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 21:54:35.085073 (+ 1212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":1106,"thread_start_us":150,"threads_started":1}
I20260620 21:54:35.087656 20113 tablet_service.cc:1467] Processing CreateTablet for tablet 0914727f08ac4f8383f66b7da5a6095e (DEFAULT_TABLE table=test-table [id=1fadec647e5645f4b09b43275835f137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:35.087662 20381 tablet_service.cc:1467] Processing CreateTablet for tablet 0914727f08ac4f8383f66b7da5a6095e (DEFAULT_TABLE table=test-table [id=1fadec647e5645f4b09b43275835f137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:35.088734 20113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0914727f08ac4f8383f66b7da5a6095e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:35.088757 20381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0914727f08ac4f8383f66b7da5a6095e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:35.089591 20247 tablet_service.cc:1467] Processing CreateTablet for tablet 0914727f08ac4f8383f66b7da5a6095e (DEFAULT_TABLE table=test-table [id=1fadec647e5645f4b09b43275835f137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:35.090751 20247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0914727f08ac4f8383f66b7da5a6095e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:35.095893 20464 tablet_bootstrap.cc:492] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed: Bootstrap starting.
I20260620 21:54:35.097937 20464 tablet_bootstrap.cc:654] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:35.097959 20466 tablet_bootstrap.cc:492] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6: Bootstrap starting.
I20260620 21:54:35.098486 20465 tablet_bootstrap.cc:492] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b: Bootstrap starting.
I20260620 21:54:35.098901 20464 log.cc:824] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:35.100674 20465 tablet_bootstrap.cc:654] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:35.100701 20466 tablet_bootstrap.cc:654] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:35.101290 20464 tablet_bootstrap.cc:492] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed: No bootstrap required, opened a new log
I20260620 21:54:35.101526 20464 ts_tablet_manager.cc:1397] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:54:35.101552 20466 log.cc:824] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:35.102013 20465 log.cc:824] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:35.103544 20466 tablet_bootstrap.cc:492] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6: No bootstrap required, opened a new log
I20260620 21:54:35.103754 20466 ts_tablet_manager.cc:1397] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 21:54:35.104099 20465 tablet_bootstrap.cc:492] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b: No bootstrap required, opened a new log
I20260620 21:54:35.104288 20465 ts_tablet_manager.cc:1397] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 21:54:35.106432 20464 raft_consensus.cc:348] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f49bc09d114d95931d7a89a8b602e6" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36443 } } peers { permanent_uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37503 } } peers { permanent_uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46715 } }
I20260620 21:54:35.106767 20464 raft_consensus.cc:374] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:35.106837 20464 raft_consensus.cc:729] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a08b4775e2a494a8ebdb57bbe4bb2ed, State: Initialized, Role: FOLLOWER
I20260620 21:54:35.107023 20466 raft_consensus.cc:348] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f49bc09d114d95931d7a89a8b602e6" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36443 } } peers { permanent_uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37503 } } peers { permanent_uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46715 } }
I20260620 21:54:35.107368 20466 raft_consensus.cc:374] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:35.107501 20466 raft_consensus.cc:729] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f49bc09d114d95931d7a89a8b602e6, State: Initialized, Role: FOLLOWER
I20260620 21:54:35.107424 20464 consensus_queue.cc:260] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f49bc09d114d95931d7a89a8b602e6" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36443 } } peers { permanent_uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37503 } } peers { permanent_uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46715 } }
I20260620 21:54:35.107985 20466 consensus_queue.cc:260] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f49bc09d114d95931d7a89a8b602e6" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36443 } } peers { permanent_uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37503 } } peers { permanent_uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46715 } }
I20260620 21:54:35.108244 20465 raft_consensus.cc:348] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f49bc09d114d95931d7a89a8b602e6" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36443 } } peers { permanent_uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37503 } } peers { permanent_uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46715 } }
I20260620 21:54:35.108512 20465 raft_consensus.cc:374] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:35.108563 20465 raft_consensus.cc:729] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9bcd15b80dc4cff80fd0f98be6a2a6b, State: Initialized, Role: FOLLOWER
I20260620 21:54:35.108614 20464 ts_tablet_manager.cc:1428] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260620 21:54:35.108937 20466 ts_tablet_manager.cc:1428] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 21:54:35.108991 20179 heartbeater.cc:503] Master 127.0.193.254:45297 was elected leader, sending a full tablet report...
I20260620 21:54:35.109009 20465 consensus_queue.cc:260] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f49bc09d114d95931d7a89a8b602e6" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36443 } } peers { permanent_uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37503 } } peers { permanent_uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46715 } }
I20260620 21:54:35.109261 20313 heartbeater.cc:503] Master 127.0.193.254:45297 was elected leader, sending a full tablet report...
I20260620 21:54:35.109689 20447 heartbeater.cc:503] Master 127.0.193.254:45297 was elected leader, sending a full tablet report...
I20260620 21:54:35.109934 20465 ts_tablet_manager.cc:1428] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 21:54:35.283097 20448 tablet.cc:2369] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:54:35.293844 20314 tablet.cc:2369] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 21:54:35.305665 20180 tablet.cc:2369] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:54:35.443965 20471 raft_consensus.cc:482] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:54:35.444185 20471 raft_consensus.cc:504] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f49bc09d114d95931d7a89a8b602e6" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36443 } } peers { permanent_uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37503 } } peers { permanent_uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46715 } }
I20260620 21:54:35.445338 20471 leader_election.cc:288] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9bcd15b80dc4cff80fd0f98be6a2a6b (127.0.193.195:37503), 3a08b4775e2a494a8ebdb57bbe4bb2ed (127.0.193.193:46715)
I20260620 21:54:35.449398 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.445865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:37753 (local address 127.0.193.193:46715)
0620 21:54:35.446038 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:35.446044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:35.446062 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:54:35.446206 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:35.446209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:35.446259 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:35.446350 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:35.446699 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.447237 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.448368 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.448802 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.448941 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.448994 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:35.449184 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 21:54:35.449256 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 21:54:35.449589 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.445640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:45067 (local address 127.0.193.195:37503)
0620 21:54:35.445785 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:35.445790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:35.445931 (+ 141us) server_negotiation.cc:407] Connection header received
0620 21:54:35.446155 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:35.446168 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:35.446213 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:35.446300 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:35.446609 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.447167 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.448475 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.448886 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.449040 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.449096 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:35.449336 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 21:54:35.449412 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:54:35.449415 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.445576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:46715 (local address 127.0.193.194:37753)
0620 21:54:35.445956 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:35.445970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:35.446078 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:35.446362 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:35.446365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:35.446404 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:35.446579 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:35.446585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.447353 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.447358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:35.448238 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:35.448247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.448938 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.448942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:35.449047 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.449075 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 21:54:35.449167 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 21:54:35.449254 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":50,"spinlock_wait_cycles":768,"thread_start_us":84,"threads_started":1}
I20260620 21:54:35.449606 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.445540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:37503 (local address 127.0.193.194:45067)
0620 21:54:35.445840 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:35.445856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:35.446028 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:35.446340 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:35.446343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:35.446359 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:35.446492 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:35.446498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.447335 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.447339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:35.448313 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:35.448323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.449046 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.449049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:35.449180 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.449226 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 21:54:35.449322 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 21:54:35.449402 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260620 21:54:35.450469 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0914727f08ac4f8383f66b7da5a6095e" candidate_uuid: "81f49bc09d114d95931d7a89a8b602e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed" is_pre_election: true
I20260620 21:54:35.450466 20401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0914727f08ac4f8383f66b7da5a6095e" candidate_uuid: "81f49bc09d114d95931d7a89a8b602e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b" is_pre_election: true
I20260620 21:54:35.450819 20133 raft_consensus.cc:2454] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f49bc09d114d95931d7a89a8b602e6 in term 0.
I20260620 21:54:35.450819 20401 raft_consensus.cc:2454] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f49bc09d114d95931d7a89a8b602e6 in term 0.
I20260620 21:54:35.451432 20200 leader_election.cc:302] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81f49bc09d114d95931d7a89a8b602e6, d9bcd15b80dc4cff80fd0f98be6a2a6b; no voters:
I20260620 21:54:35.451792 20471 raft_consensus.cc:2781] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:54:35.451895 20471 raft_consensus.cc:482] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:54:35.451946 20471 raft_consensus.cc:3037] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:35.453200 20471 raft_consensus.cc:504] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f49bc09d114d95931d7a89a8b602e6" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36443 } } peers { permanent_uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37503 } } peers { permanent_uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46715 } }
I20260620 21:54:35.453652 20471 leader_election.cc:288] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [CANDIDATE]: Term 1 election: Requested vote from peers d9bcd15b80dc4cff80fd0f98be6a2a6b (127.0.193.195:37503), 3a08b4775e2a494a8ebdb57bbe4bb2ed (127.0.193.193:46715)
I20260620 21:54:35.454157 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0914727f08ac4f8383f66b7da5a6095e" candidate_uuid: "81f49bc09d114d95931d7a89a8b602e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed"
I20260620 21:54:35.454161 20401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0914727f08ac4f8383f66b7da5a6095e" candidate_uuid: "81f49bc09d114d95931d7a89a8b602e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b"
I20260620 21:54:35.454282 20401 raft_consensus.cc:3037] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:35.454319 20133 raft_consensus.cc:3037] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:35.455499 20401 raft_consensus.cc:2454] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f49bc09d114d95931d7a89a8b602e6 in term 1.
I20260620 21:54:35.455619 20133 raft_consensus.cc:2454] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f49bc09d114d95931d7a89a8b602e6 in term 1.
I20260620 21:54:35.455930 20200 leader_election.cc:302] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81f49bc09d114d95931d7a89a8b602e6, d9bcd15b80dc4cff80fd0f98be6a2a6b; no voters:
I20260620 21:54:35.456166 20471 raft_consensus.cc:2781] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:35.456413 20471 raft_consensus.cc:686] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [term 1 LEADER]: Becoming Leader. State: Replica: 81f49bc09d114d95931d7a89a8b602e6, State: Running, Role: LEADER
I20260620 21:54:35.456799 20471 consensus_queue.cc:237] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f49bc09d114d95931d7a89a8b602e6" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36443 } } peers { permanent_uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37503 } } peers { permanent_uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46715 } }
I20260620 21:54:35.460044 19988 catalog_manager.cc:5495] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81f49bc09d114d95931d7a89a8b602e6 (127.0.193.194). New cstate: current_term: 1 leader_uuid: "81f49bc09d114d95931d7a89a8b602e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81f49bc09d114d95931d7a89a8b602e6" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 36443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46715 } health_report { overall_health: UNKNOWN } } }
I20260620 21:54:35.539623 20459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.535604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.0.193.194:36443)
0620 21:54:35.535764 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:35.535769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:35.535810 (+ 41us) server_negotiation.cc:407] Connection header received
0620 21:54:35.535885 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:35.535888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:35.535947 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:35.536037 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:54:35.536291 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.536869 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.537746 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.538046 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.538172 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.538478 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:54:35.539017 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:54:35.539120 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:35.539346 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 21:54:35.539409 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 21:54:35.546208 20401 raft_consensus.cc:1261] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b [term 1 FOLLOWER]: Refusing update from remote peer 81f49bc09d114d95931d7a89a8b602e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:54:35.546219 20133 raft_consensus.cc:1261] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed [term 1 FOLLOWER]: Refusing update from remote peer 81f49bc09d114d95931d7a89a8b602e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:54:35.547017 20475 consensus_queue.cc:1035] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9bcd15b80dc4cff80fd0f98be6a2a6b" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 37503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:54:35.547228 20471 consensus_queue.cc:1035] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a08b4775e2a494a8ebdb57bbe4bb2ed" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:54:35.556381 20478 mvcc.cc:205] Tried to move back new op lower bound from 7299041179828613120 to 7299041179473764352. Current Snapshot: MvccSnapshot[applied={T|T < 7299041179828613120}]
I20260620 21:54:35.567636 19988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:54:35.567819 19988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:54:35.570540 19988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34bd81b6c6b64d9c9539cdbf20c4cddd: Sending DeleteTablet for 3 replicas of tablet 0914727f08ac4f8383f66b7da5a6095e
I20260620 21:54:35.571539 20247 tablet_service.cc:1514] Processing DeleteTablet for tablet 0914727f08ac4f8383f66b7da5a6095e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:35 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:39338
I20260620 21:54:35.571537 20381 tablet_service.cc:1514] Processing DeleteTablet for tablet 0914727f08ac4f8383f66b7da5a6095e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:35 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:55010
I20260620 21:54:35.572116 20487 tablet_replica.cc:330] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6: stopping tablet replica
I20260620 21:54:35.572088 20113 tablet_service.cc:1514] Processing DeleteTablet for tablet 0914727f08ac4f8383f66b7da5a6095e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:35 UTC) from {username='kudu', principal='kudu/127.0.193.254@KRBTEST.COM'} at 127.0.0.1:51732
I20260620 21:54:35.572520 20489 tablet_replica.cc:330] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed: stopping tablet replica
I20260620 21:54:35.572623 20487 raft_consensus.cc:2229] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:54:35.572827 20489 raft_consensus.cc:2229] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:54:35.572958 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 20050
I20260620 21:54:35.573159 20487 raft_consensus.cc:2258] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:54:35.573204 20489 raft_consensus.cc:2258] T 0914727f08ac4f8383f66b7da5a6095e P 3a08b4775e2a494a8ebdb57bbe4bb2ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:54:35.573662 20488 tablet_replica.cc:330] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b: stopping tablet replica
I20260620 21:54:35.574034 20488 raft_consensus.cc:2229] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:54:35.574415 20488 raft_consensus.cc:2258] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:54:35.575467 20487 ts_tablet_manager.cc:1905] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:54:35.575963 20488 ts_tablet_manager.cc:1905] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:54:35.578691 20488 ts_tablet_manager.cc:1918] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:54:35.578873 20488 log.cc:1196] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/0914727f08ac4f8383f66b7da5a6095e
I20260620 21:54:35.579097 20487 ts_tablet_manager.cc:1918] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:54:35.579226 20487 log.cc:1196] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781992210835692-775-0/minicluster-data/ts-1/wal/wals/0914727f08ac4f8383f66b7da5a6095e
I20260620 21:54:35.579420 20488 ts_tablet_manager.cc:1939] T 0914727f08ac4f8383f66b7da5a6095e P d9bcd15b80dc4cff80fd0f98be6a2a6b: Deleting consensus metadata
I20260620 21:54:35.579555 20487 ts_tablet_manager.cc:1939] T 0914727f08ac4f8383f66b7da5a6095e P 81f49bc09d114d95931d7a89a8b602e6: Deleting consensus metadata
I20260620 21:54:35.580862 19972 catalog_manager.cc:4841] TS d9bcd15b80dc4cff80fd0f98be6a2a6b (127.0.193.195:37503): tablet 0914727f08ac4f8383f66b7da5a6095e (table test-table [id=1fadec647e5645f4b09b43275835f137]) successfully deleted
I20260620 21:54:35.581103 19972 catalog_manager.cc:4841] TS 81f49bc09d114d95931d7a89a8b602e6 (127.0.193.194:36443): tablet 0914727f08ac4f8383f66b7da5a6095e (table test-table [id=1fadec647e5645f4b09b43275835f137]) successfully deleted
W20260620 21:54:35.583287 19973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:46715 (error 108)
I20260620 21:54:35.583484 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 20184
I20260620 21:54:35.584895 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.584546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:46715 (local address 127.0.0.1:37684)
0620 21:54:35.584690 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:35.584774 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 21:54:35.585088 19973 catalog_manager.cc:4584] TS 3a08b4775e2a494a8ebdb57bbe4bb2ed (127.0.193.193:46715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0914727f08ac4f8383f66b7da5a6095e: Network error: Client connection negotiation failed: client connection to 127.0.193.193:46715: connect: Connection refused (error 111)
I20260620 21:54:35.589313 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 20318
I20260620 21:54:35.594421 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 19956
2026-06-20T21:54:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2756 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T21:54:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:35Z Disabled control of system clock
I20260620 21:54:35.628345 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:41977
--webserver_interface=127.0.193.254
--webserver_port=0
--builtin_ntp_servers=127.0.193.212:34183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:41977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:54:35.750700 20494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:35.751051 20494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:35.751092 20494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:35.755553 20494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:35.755633 20494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:35.755648 20494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:35.755666 20494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:35.755682 20494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:35.760699 20494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:41977
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:41977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:35.761854 20494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:35.763242 20494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:35.773154 20500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:35.773198 20499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:35.773277 20502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:35.773481 20494 server_base.cc:1034] running on GCE node
I20260620 21:54:35.774173 20494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:35.775527 20494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:35.776728 20494 hybrid_clock.cc:648] HybridClock initialized: now 1781992475776697 us; error 53 us; skew 500 ppm
I20260620 21:54:35.782056 20494 webserver.cc:460] Webserver started at http://127.0.193.254:42695/ using document root <none> and password file <none>
I20260620 21:54:35.782881 20494 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:35.782958 20494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:35.783269 20494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:35.785527 20494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "f5b77a9f129f46f280bf28ecc45e7dba"
format_stamp: "Formatted at 2026-06-20 21:54:35 on dist-test-slave-kk5c"
server_key: "b361e3f96e505b624e5cd91232423129"
server_key_iv: "291a7f385bf2ffa8f96fb3d424f36376"
server_key_version: "encryptionkey@0"
I20260620 21:54:35.786116 20494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "f5b77a9f129f46f280bf28ecc45e7dba"
format_stamp: "Formatted at 2026-06-20 21:54:35 on dist-test-slave-kk5c"
server_key: "b361e3f96e505b624e5cd91232423129"
server_key_iv: "291a7f385bf2ffa8f96fb3d424f36376"
server_key_version: "encryptionkey@0"
I20260620 21:54:35.790411 20494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:54:35.793139 20508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:35.794544 20494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:54:35.794703 20494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "f5b77a9f129f46f280bf28ecc45e7dba"
format_stamp: "Formatted at 2026-06-20 21:54:35 on dist-test-slave-kk5c"
server_key: "b361e3f96e505b624e5cd91232423129"
server_key_iv: "291a7f385bf2ffa8f96fb3d424f36376"
server_key_version: "encryptionkey@0"
I20260620 21:54:35.794829 20494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:35.825666 20494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:35.826467 20494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:35.826689 20494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:35.834187 20494 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:41977
I20260620 21:54:35.834233 20560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:41977 every 8 connection(s)
I20260620 21:54:35.835292 20494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:35.836244 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 20494
I20260620 21:54:35.836403 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:35.836735 775 external_mini_cluster.cc:1428] Setting key 994bc9d3447a71486476f33818681b03
I20260620 21:54:35.839459 20561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:35.846256 20561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba: Bootstrap starting.
I20260620 21:54:35.847440 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:35.838447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.0.193.254:41977)
0620 21:54:35.839285 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:35.839303 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:35.839344 (+ 41us) server_negotiation.cc:407] Connection header received
0620 21:54:35.840115 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:35.840135 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:35.840589 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:35.840911 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:35.841296 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.842380 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.843675 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:35.844199 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:35.844329 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:35.845453 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:35.845459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:35.845472 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:35.845897 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:35.845960 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:35.845981 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:35.846073 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:35.846293 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 21:54:35.846609 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":158,"threads_started":1}
I20260620 21:54:35.848718 20561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:35.849532 20561 log.cc:824] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:35.851999 20561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba: No bootstrap required, opened a new log
I20260620 21:54:35.856019 20561 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b77a9f129f46f280bf28ecc45e7dba" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41977 } }
I20260620 21:54:35.856285 20561 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:35.856354 20561 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b77a9f129f46f280bf28ecc45e7dba, State: Initialized, Role: FOLLOWER
I20260620 21:54:35.856818 20561 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b77a9f129f46f280bf28ecc45e7dba" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41977 } }
I20260620 21:54:35.856958 20561 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:35.857064 20561 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:35.857188 20561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:35.858255 20561 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b77a9f129f46f280bf28ecc45e7dba" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41977 } }
I20260620 21:54:35.858637 20561 leader_election.cc:302] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5b77a9f129f46f280bf28ecc45e7dba; no voters:
I20260620 21:54:35.859074 20561 leader_election.cc:288] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:35.859153 20566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:35.859526 20566 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [term 1 LEADER]: Becoming Leader. State: Replica: f5b77a9f129f46f280bf28ecc45e7dba, State: Running, Role: LEADER
I20260620 21:54:35.859841 20566 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b77a9f129f46f280bf28ecc45e7dba" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41977 } }
I20260620 21:54:35.860603 20561 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:35.861603 20568 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5b77a9f129f46f280bf28ecc45e7dba. Latest consensus state: current_term: 1 leader_uuid: "f5b77a9f129f46f280bf28ecc45e7dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b77a9f129f46f280bf28ecc45e7dba" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41977 } } }
I20260620 21:54:35.861737 20568 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:35.862049 20567 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5b77a9f129f46f280bf28ecc45e7dba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b77a9f129f46f280bf28ecc45e7dba" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 41977 } } }
I20260620 21:54:35.862139 20567 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:35.862279 20575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:35.865029 20575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:35.871443 20575 catalog_manager.cc:1347] Generated new cluster ID: 8721d2cf7caf437383d6bd90ce2eb578
I20260620 21:54:35.871529 20575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:35.886111 20575 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:35.887477 20575 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:35.909619 20575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba: Generated new TSK 0
I20260620 21:54:35.910625 20575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:35.916445 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.193:0
--local_ip_for_outbound_sockets=127.0.193.193
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41977
--builtin_ntp_servers=127.0.193.212:34183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 21:54:36.041229 20585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:36.041566 20585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:36.041638 20585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:36.045854 20585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:36.045958 20585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:36.046058 20585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.193
I20260620 21:54:36.051222 20585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.193.193
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.193
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:36.052465 20585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:36.053903 20585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:36.064455 20590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:36.064541 20593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:36.064463 20591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:36.064849 20585 server_base.cc:1034] running on GCE node
I20260620 21:54:36.065312 20585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:36.066053 20585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:36.067281 20585 hybrid_clock.cc:648] HybridClock initialized: now 1781992476067257 us; error 33 us; skew 500 ppm
I20260620 21:54:36.069653 20585 webserver.cc:460] Webserver started at http://127.0.193.193:35105/ using document root <none> and password file <none>
I20260620 21:54:36.070324 20585 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:36.070444 20585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:36.070721 20585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:36.073341 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/data/instance:
uuid: "c7918819891b4d61b00a08123b35898d"
format_stamp: "Formatted at 2026-06-20 21:54:36 on dist-test-slave-kk5c"
server_key: "f882fbca41223a079e9b63fd31a6c51b"
server_key_iv: "4fab15d6976651a9491f4a2a8aa1b96a"
server_key_version: "encryptionkey@0"
I20260620 21:54:36.074030 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/wal/instance:
uuid: "c7918819891b4d61b00a08123b35898d"
format_stamp: "Formatted at 2026-06-20 21:54:36 on dist-test-slave-kk5c"
server_key: "f882fbca41223a079e9b63fd31a6c51b"
server_key_iv: "4fab15d6976651a9491f4a2a8aa1b96a"
server_key_version: "encryptionkey@0"
I20260620 21:54:36.078198 20585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 21:54:36.081064 20599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:36.082513 20585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:54:36.082674 20585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/wal
uuid: "c7918819891b4d61b00a08123b35898d"
format_stamp: "Formatted at 2026-06-20 21:54:36 on dist-test-slave-kk5c"
server_key: "f882fbca41223a079e9b63fd31a6c51b"
server_key_iv: "4fab15d6976651a9491f4a2a8aa1b96a"
server_key_version: "encryptionkey@0"
I20260620 21:54:36.082820 20585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:36.111349 20585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:36.112121 20585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:36.112347 20585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:36.112984 20585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:36.114025 20585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:36.114121 20585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:36.114189 20585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:36.114208 20585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:36.125334 20585 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.193:38141
I20260620 21:54:36.125391 20712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.193:38141 every 8 connection(s)
I20260620 21:54:36.126410 20585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/data/info.pb
I20260620 21:54:36.134656 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 20585
I20260620 21:54:36.134765 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-0/wal/instance
I20260620 21:54:36.135107 775 external_mini_cluster.cc:1428] Setting key d2a8d1e06b08102db4b149d71b8cef31
I20260620 21:54:36.136457 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.128419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.193:45083 (local address 127.0.193.254:41977)
0620 21:54:36.128600 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:36.128606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:36.129736 (+ 1130us) server_negotiation.cc:407] Connection header received
0620 21:54:36.131099 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:36.131102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:36.131160 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:36.131312 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:36.132259 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.132855 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.134115 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.134366 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.134513 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.135894 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:36.135897 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:36.135900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:36.135921 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:36.135943 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:36.135952 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:36.136031 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:36.136207 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 21:54:36.136299 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 21:54:36.137574 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.194:0
--local_ip_for_outbound_sockets=127.0.193.194
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41977
--builtin_ntp_servers=127.0.193.212:34183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:54:36.137914 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.128893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41977 (local address 127.0.193.193:45083)
0620 21:54:36.129568 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:36.129609 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:36.130888 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:36.131517 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:36.131525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:36.131610 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:36.132114 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.132127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.133041 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.133044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.133984 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.133994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.134562 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.134566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.134814 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.135571 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:54:36.135576 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:36.135634 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:54:36.135725 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:36.136046 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:36.136056 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:36.136061 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:54:36.136196 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 21:54:36.136699 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":121,"threads_started":1}
I20260620 21:54:36.139457 20713 heartbeater.cc:347] Connected to a master server at 127.0.193.254:41977
I20260620 21:54:36.139757 20713 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:36.140480 20713 heartbeater.cc:511] Master 127.0.193.254:41977 requested a full tablet report, sending...
I20260620 21:54:36.142357 20525 ts_manager.cc:194] Registered new tserver with Master: c7918819891b4d61b00a08123b35898d (127.0.193.193:38141)
I20260620 21:54:36.143767 20525 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.193.193:45083
W20260620 21:54:36.252070 20716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:36.252403 20716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:36.252445 20716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:36.256714 20716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:36.256786 20716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:36.256851 20716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.194
I20260620 21:54:36.262442 20716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.193.194
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.194
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:36.263703 20716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:36.265156 20716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:36.275533 20722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:36.275508 20724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:36.275511 20721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:36.275942 20716 server_base.cc:1034] running on GCE node
I20260620 21:54:36.276386 20716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:36.277108 20716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:36.278304 20716 hybrid_clock.cc:648] HybridClock initialized: now 1781992476278267 us; error 54 us; skew 500 ppm
I20260620 21:54:36.280439 20716 webserver.cc:460] Webserver started at http://127.0.193.194:38587/ using document root <none> and password file <none>
I20260620 21:54:36.281065 20716 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:36.281131 20716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:36.281409 20716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:36.283643 20716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/data/instance:
uuid: "f1ad5c2f0e234f90b031337e6d18cde0"
format_stamp: "Formatted at 2026-06-20 21:54:36 on dist-test-slave-kk5c"
server_key: "509bceacb65a0800dccf4e6a4e21e5c1"
server_key_iv: "7effebeeca0203493ccbe1641fc200d4"
server_key_version: "encryptionkey@0"
I20260620 21:54:36.284265 20716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/wal/instance:
uuid: "f1ad5c2f0e234f90b031337e6d18cde0"
format_stamp: "Formatted at 2026-06-20 21:54:36 on dist-test-slave-kk5c"
server_key: "509bceacb65a0800dccf4e6a4e21e5c1"
server_key_iv: "7effebeeca0203493ccbe1641fc200d4"
server_key_version: "encryptionkey@0"
I20260620 21:54:36.288941 20716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:54:36.292106 20730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:36.293599 20716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 21:54:36.293776 20716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/wal
uuid: "f1ad5c2f0e234f90b031337e6d18cde0"
format_stamp: "Formatted at 2026-06-20 21:54:36 on dist-test-slave-kk5c"
server_key: "509bceacb65a0800dccf4e6a4e21e5c1"
server_key_iv: "7effebeeca0203493ccbe1641fc200d4"
server_key_version: "encryptionkey@0"
I20260620 21:54:36.293915 20716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:36.305974 20716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:36.306706 20716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:36.306932 20716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:36.307615 20716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:36.308617 20716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:36.308687 20716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:36.308760 20716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:36.308804 20716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:36.317844 20716 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.194:41643
I20260620 21:54:36.317865 20843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.194:41643 every 8 connection(s)
I20260620 21:54:36.318786 20716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/data/info.pb
I20260620 21:54:36.326223 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 20716
I20260620 21:54:36.326360 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-1/wal/instance
I20260620 21:54:36.326689 775 external_mini_cluster.cc:1428] Setting key 7ab1e4869c70222af6e56440640bcfeb
I20260620 21:54:36.328585 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.320817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.194:42259 (local address 127.0.193.254:41977)
0620 21:54:36.320987 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:36.320991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:36.322077 (+ 1086us) server_negotiation.cc:407] Connection header received
0620 21:54:36.323489 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:36.323493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:36.323544 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:36.323628 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:36.324603 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.325199 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.326456 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.326689 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.326810 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.328060 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:36.328063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:36.328066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:36.328084 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:36.328103 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:36.328112 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:36.328198 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:36.328366 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 21:54:36.328439 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 21:54:36.329046 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.193.195:0
--local_ip_for_outbound_sockets=127.0.193.195
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41977
--builtin_ntp_servers=127.0.193.212:34183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 21:54:36.329708 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.321174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41977 (local address 127.0.193.194:42259)
0620 21:54:36.321871 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:36.321948 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:36.323253 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:36.323806 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:36.323813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:36.323932 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:36.324433 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.324447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.325334 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.325337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.326235 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.326243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.326857 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.326861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.327048 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.327759 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:54:36.327761 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:36.327806 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:54:36.327877 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:36.328200 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:36.328212 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:36.328221 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 21:54:36.328363 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 21:54:36.328681 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":117,"threads_started":1}
I20260620 21:54:36.331243 20844 heartbeater.cc:347] Connected to a master server at 127.0.193.254:41977
I20260620 21:54:36.331540 20844 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:36.332177 20844 heartbeater.cc:511] Master 127.0.193.254:41977 requested a full tablet report, sending...
I20260620 21:54:36.333475 20525 ts_manager.cc:194] Registered new tserver with Master: f1ad5c2f0e234f90b031337e6d18cde0 (127.0.193.194:41643)
I20260620 21:54:36.334218 20525 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.193.194:42259
W20260620 21:54:36.455387 20847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:36.455708 20847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:36.455763 20847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:36.460546 20847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:36.460621 20847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:36.460687 20847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.193.195
I20260620 21:54:36.466427 20847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:34183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.193.195
--webserver_port=0
--tserver_master_addrs=127.0.193.254:41977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.193.195
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:36.467617 20847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:36.469076 20847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:36.479492 20855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:36.479492 20853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:36.479751 20852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:36.479769 20847 server_base.cc:1034] running on GCE node
I20260620 21:54:36.480275 20847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:36.481032 20847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:36.482205 20847 hybrid_clock.cc:648] HybridClock initialized: now 1781992476482177 us; error 42 us; skew 500 ppm
I20260620 21:54:36.484241 20847 webserver.cc:460] Webserver started at http://127.0.193.195:40583/ using document root <none> and password file <none>
I20260620 21:54:36.484869 20847 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:36.484963 20847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:36.485193 20847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:36.487834 20847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/data/instance:
uuid: "09d5905b9343414bb1598034b581dce2"
format_stamp: "Formatted at 2026-06-20 21:54:36 on dist-test-slave-kk5c"
server_key: "c934797153e9e466eadc2c99e7a8a94c"
server_key_iv: "404601915839c2da19cbae1558ea1aba"
server_key_version: "encryptionkey@0"
I20260620 21:54:36.488497 20847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/wal/instance:
uuid: "09d5905b9343414bb1598034b581dce2"
format_stamp: "Formatted at 2026-06-20 21:54:36 on dist-test-slave-kk5c"
server_key: "c934797153e9e466eadc2c99e7a8a94c"
server_key_iv: "404601915839c2da19cbae1558ea1aba"
server_key_version: "encryptionkey@0"
I20260620 21:54:36.492493 20847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:54:36.495136 20861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:36.496519 20847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:36.496649 20847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/wal
uuid: "09d5905b9343414bb1598034b581dce2"
format_stamp: "Formatted at 2026-06-20 21:54:36 on dist-test-slave-kk5c"
server_key: "c934797153e9e466eadc2c99e7a8a94c"
server_key_iv: "404601915839c2da19cbae1558ea1aba"
server_key_version: "encryptionkey@0"
I20260620 21:54:36.496783 20847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:36.523114 20847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:36.523938 20847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:36.524143 20847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:36.524767 20847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 21:54:36.525727 20847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 21:54:36.525813 20847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:36.525885 20847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 21:54:36.525935 20847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:36.536299 20847 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.195:45535
I20260620 21:54:36.536340 20974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.195:45535 every 8 connection(s)
I20260620 21:54:36.537298 20847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/data/info.pb
I20260620 21:54:36.538089 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 20847
I20260620 21:54:36.538190 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/wal/instance
I20260620 21:54:36.538515 775 external_mini_cluster.cc:1428] Setting key e31e535b79c3ce4cc0f606b3cd828366
I20260620 21:54:36.547909 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.539657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:40919 (local address 127.0.193.254:41977)
0620 21:54:36.539853 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:36.539858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:36.541302 (+ 1444us) server_negotiation.cc:407] Connection header received
0620 21:54:36.542744 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:36.542747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:36.542793 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:36.542873 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:36.543821 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.544355 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.545648 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.545868 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.546022 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.547324 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:36.547327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:36.547335 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:36.547353 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:36.547375 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:36.547384 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:36.547461 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:36.547684 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 21:54:36.547745 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 21:54:36.548808 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.540208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:41977 (local address 127.0.193.195:40919)
0620 21:54:36.541126 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:36.541171 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:36.542537 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:36.543040 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:36.543047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:36.543134 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:54:36.543676 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.543689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.544492 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.544495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.545510 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.545521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.546106 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.546109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.546280 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.546984 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 21:54:36.546985 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:54:36.547026 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 21:54:36.547100 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:54:36.547499 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:54:36.547507 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:54:36.547512 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 21:54:36.547634 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 21:54:36.547879 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":147,"threads_started":1}
I20260620 21:54:36.550231 20975 heartbeater.cc:347] Connected to a master server at 127.0.193.254:41977
I20260620 21:54:36.550550 20975 heartbeater.cc:464] Registering TS with master...
I20260620 21:54:36.551451 20975 heartbeater.cc:511] Master 127.0.193.254:41977 requested a full tablet report, sending...
I20260620 21:54:36.552688 20525 ts_manager.cc:194] Registered new tserver with Master: 09d5905b9343414bb1598034b581dce2 (127.0.193.195:45535)
I20260620 21:54:36.553117 775 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 21:54:36.553395 20525 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.193.195:40919
I20260620 21:54:36.559132 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.555251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.0.193.254:41977)
0620 21:54:36.555374 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:36.555379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:36.555554 (+ 175us) server_negotiation.cc:407] Connection header received
0620 21:54:36.555652 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:36.555654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:36.555690 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:36.555754 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:36.556008 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.556489 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.557414 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.557633 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.557714 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.558647 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:36.558650 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:36.558653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:36.558666 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:36.558688 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:36.558696 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:36.558779 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:36.558864 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 21:54:36.558915 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:54:36.562259 20525 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 21:54:36.565774 20525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 21:54:36.581709 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.575360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:41643 (local address 127.0.0.1:32950)
0620 21:54:36.575835 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:36.575899 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:36.576086 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:36.576794 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:36.576801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:36.576829 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:36.577004 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.577013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.578645 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.578652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.580009 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.580018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.581038 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.581042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.581214 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.581260 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 21:54:36.581394 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 21:54:36.581534 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":29,"thread_start_us":117,"threads_started":1}
I20260620 21:54:36.582412 20987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.575494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32950 (local address 127.0.193.194:41643)
0620 21:54:36.575843 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:36.575856 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:36.576013 (+ 157us) server_negotiation.cc:407] Connection header received
0620 21:54:36.576262 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:36.576268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:36.576522 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:36.577192 (+ 670us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:36.577369 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.578527 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.580155 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.580901 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.581038 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.581273 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:36.582077 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 21:54:36.582200 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":132,"threads_started":1}
I20260620 21:54:36.582557 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.576230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:38141 (local address 127.0.0.1:51480)
0620 21:54:36.576472 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:36.576490 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:36.576584 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:36.577843 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:36.577849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:36.577871 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:36.577988 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.577996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.579887 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.579891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.581110 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.581122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.582140 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.582144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.582224 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.582248 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 21:54:36.582291 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 21:54:36.582392 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":103,"threads_started":1}
I20260620 21:54:36.583084 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.576752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.195:45535 (local address 127.0.0.1:36252)
0620 21:54:36.577005 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:36.577020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:36.577109 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:36.578015 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:36.578018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:36.578037 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:36.578135 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.578143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.579672 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.579676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.581266 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.581275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.582377 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.582381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.582465 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.582776 (+ 311us) client_negotiation.cc:769] Sending connection context
0620 21:54:36.582869 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 21:54:36.582949 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":50,"thread_start_us":92,"threads_started":1}
I20260620 21:54:36.583273 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.576940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.0.193.195:45535)
0620 21:54:36.577425 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:36.577442 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:36.577466 (+ 24us) server_negotiation.cc:407] Connection header received
0620 21:54:36.577551 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:36.577557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:36.577726 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:36.577906 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:36.578268 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.579514 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.581379 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.582246 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.582368 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.582578 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:36.582942 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 21:54:36.583080 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":157,"threads_started":1}
I20260620 21:54:36.584290 20989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.576441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.0.193.193:38141)
0620 21:54:36.577359 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:36.577372 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:36.577397 (+ 25us) server_negotiation.cc:407] Connection header received
0620 21:54:36.577477 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:36.577482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:36.577676 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:36.577851 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:36.578139 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.579759 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.581303 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.581989 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.583741 (+ 1752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.583950 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:36.584005 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 21:54:36.584112 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":252,"threads_started":1}
I20260620 21:54:36.585988 20778 tablet_service.cc:1467] Processing CreateTablet for tablet 131b90b022434163b962b157e5a18a43 (DEFAULT_TABLE table=test-table [id=fdebc216b50f4d829b9010323bfa5260]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:36.586103 20908 tablet_service.cc:1467] Processing CreateTablet for tablet 131b90b022434163b962b157e5a18a43 (DEFAULT_TABLE table=test-table [id=fdebc216b50f4d829b9010323bfa5260]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:36.587242 20908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 131b90b022434163b962b157e5a18a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:36.587260 20778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 131b90b022434163b962b157e5a18a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:36.586995 20647 tablet_service.cc:1467] Processing CreateTablet for tablet 131b90b022434163b962b157e5a18a43 (DEFAULT_TABLE table=test-table [id=fdebc216b50f4d829b9010323bfa5260]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 21:54:36.587962 20647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 131b90b022434163b962b157e5a18a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:36.594117 20992 tablet_bootstrap.cc:492] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0: Bootstrap starting.
I20260620 21:54:36.595551 20993 tablet_bootstrap.cc:492] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d: Bootstrap starting.
I20260620 21:54:36.596604 20992 tablet_bootstrap.cc:654] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:36.597296 20993 tablet_bootstrap.cc:654] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:36.597447 20992 log.cc:824] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:36.598098 20993 log.cc:824] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:36.598338 20994 tablet_bootstrap.cc:492] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2: Bootstrap starting.
I20260620 21:54:36.600121 20993 tablet_bootstrap.cc:492] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d: No bootstrap required, opened a new log
I20260620 21:54:36.600339 20993 ts_tablet_manager.cc:1397] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 21:54:36.600363 20992 tablet_bootstrap.cc:492] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0: No bootstrap required, opened a new log
I20260620 21:54:36.600562 20992 ts_tablet_manager.cc:1397] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260620 21:54:36.603977 20993 raft_consensus.cc:348] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad5c2f0e234f90b031337e6d18cde0" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41643 } } peers { permanent_uuid: "c7918819891b4d61b00a08123b35898d" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38141 } } peers { permanent_uuid: "09d5905b9343414bb1598034b581dce2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45535 } }
I20260620 21:54:36.604226 20993 raft_consensus.cc:374] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:36.604269 20993 raft_consensus.cc:729] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7918819891b4d61b00a08123b35898d, State: Initialized, Role: FOLLOWER
I20260620 21:54:36.604184 20992 raft_consensus.cc:348] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad5c2f0e234f90b031337e6d18cde0" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41643 } } peers { permanent_uuid: "c7918819891b4d61b00a08123b35898d" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38141 } } peers { permanent_uuid: "09d5905b9343414bb1598034b581dce2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45535 } }
I20260620 21:54:36.604511 20992 raft_consensus.cc:374] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:36.604593 20992 raft_consensus.cc:729] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ad5c2f0e234f90b031337e6d18cde0, State: Initialized, Role: FOLLOWER
I20260620 21:54:36.604821 20993 consensus_queue.cc:260] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad5c2f0e234f90b031337e6d18cde0" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41643 } } peers { permanent_uuid: "c7918819891b4d61b00a08123b35898d" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38141 } } peers { permanent_uuid: "09d5905b9343414bb1598034b581dce2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45535 } }
I20260620 21:54:36.605059 20992 consensus_queue.cc:260] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad5c2f0e234f90b031337e6d18cde0" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41643 } } peers { permanent_uuid: "c7918819891b4d61b00a08123b35898d" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38141 } } peers { permanent_uuid: "09d5905b9343414bb1598034b581dce2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45535 } }
I20260620 21:54:36.605937 20993 ts_tablet_manager.cc:1428] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 21:54:36.606081 20992 ts_tablet_manager.cc:1428] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260620 21:54:36.606474 20844 heartbeater.cc:503] Master 127.0.193.254:41977 was elected leader, sending a full tablet report...
I20260620 21:54:36.606789 20994 tablet_bootstrap.cc:654] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:36.607798 20713 heartbeater.cc:503] Master 127.0.193.254:41977 was elected leader, sending a full tablet report...
I20260620 21:54:36.607812 20994 log.cc:824] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:36.609848 20994 tablet_bootstrap.cc:492] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2: No bootstrap required, opened a new log
I20260620 21:54:36.610081 20994 ts_tablet_manager.cc:1397] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260620 21:54:36.613826 20994 raft_consensus.cc:348] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad5c2f0e234f90b031337e6d18cde0" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41643 } } peers { permanent_uuid: "c7918819891b4d61b00a08123b35898d" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38141 } } peers { permanent_uuid: "09d5905b9343414bb1598034b581dce2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45535 } }
I20260620 21:54:36.614135 20994 raft_consensus.cc:374] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:36.614199 20994 raft_consensus.cc:729] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d5905b9343414bb1598034b581dce2, State: Initialized, Role: FOLLOWER
I20260620 21:54:36.614724 20994 consensus_queue.cc:260] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad5c2f0e234f90b031337e6d18cde0" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41643 } } peers { permanent_uuid: "c7918819891b4d61b00a08123b35898d" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38141 } } peers { permanent_uuid: "09d5905b9343414bb1598034b581dce2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45535 } }
I20260620 21:54:36.615486 20975 heartbeater.cc:503] Master 127.0.193.254:41977 was elected leader, sending a full tablet report...
I20260620 21:54:36.615923 20994 ts_tablet_manager.cc:1428] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260620 21:54:36.627723 20714 tablet.cc:2369] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:54:36.663595 21000 raft_consensus.cc:482] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 21:54:36.663894 21000 raft_consensus.cc:504] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad5c2f0e234f90b031337e6d18cde0" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41643 } } peers { permanent_uuid: "c7918819891b4d61b00a08123b35898d" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38141 } } peers { permanent_uuid: "09d5905b9343414bb1598034b581dce2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45535 } }
I20260620 21:54:36.665053 21000 leader_election.cc:288] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1ad5c2f0e234f90b031337e6d18cde0 (127.0.193.194:41643), c7918819891b4d61b00a08123b35898d (127.0.193.193:38141)
I20260620 21:54:36.670377 20989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.666054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:53335 (local address 127.0.193.193:38141)
0620 21:54:36.666192 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:36.666196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:36.666217 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:54:36.666264 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:36.666266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:36.666303 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:36.666398 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:36.666659 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.667404 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.669049 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.669665 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.669850 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.669903 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:36.670168 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 21:54:36.670244 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:54:36.670384 20987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.665479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.193.195:54595 (local address 127.0.193.194:41643)
0620 21:54:36.665727 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:36.665731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:36.665747 (+ 16us) server_negotiation.cc:407] Connection header received
0620 21:54:36.665952 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:36.665956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:36.666012 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:36.666090 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 21:54:36.666402 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.667212 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.669049 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.669664 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.669795 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.669849 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:36.670166 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 21:54:36.670285 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260620 21:54:36.670428 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.665677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:38141 (local address 127.0.193.195:53335)
0620 21:54:36.666023 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:36.666035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:36.666145 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:36.666433 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:36.666436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:36.666453 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:36.666540 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.666544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.667597 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.667600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.668896 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.668908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.669845 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.669848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.669980 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.670037 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 21:54:36.670132 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 21:54:36.670235 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":161,"thread_start_us":136,"threads_started":1}
I20260620 21:54:36.670440 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.665336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:41643 (local address 127.0.193.195:54595)
0620 21:54:36.665507 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:36.665643 (+ 136us) client_negotiation.cc:174] Beginning negotiation
0620 21:54:36.665825 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:54:36.666129 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:54:36.666132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:54:36.666154 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 21:54:36.666287 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.666293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.667343 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.667345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.668896 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:54:36.668907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.669801 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.669806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:54:36.669999 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.670049 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 21:54:36.670120 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 21:54:36.670237 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":124,"spinlock_wait_cycles":2432}
I20260620 21:54:36.671530 20798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131b90b022434163b962b157e5a18a43" candidate_uuid: "09d5905b9343414bb1598034b581dce2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ad5c2f0e234f90b031337e6d18cde0" is_pre_election: true
I20260620 21:54:36.671564 20667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131b90b022434163b962b157e5a18a43" candidate_uuid: "09d5905b9343414bb1598034b581dce2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7918819891b4d61b00a08123b35898d" is_pre_election: true
I20260620 21:54:36.671887 20667 raft_consensus.cc:2454] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09d5905b9343414bb1598034b581dce2 in term 0.
I20260620 21:54:36.671880 20798 raft_consensus.cc:2454] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09d5905b9343414bb1598034b581dce2 in term 0.
I20260620 21:54:36.672406 20864 leader_election.cc:302] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d5905b9343414bb1598034b581dce2, c7918819891b4d61b00a08123b35898d; no voters:
I20260620 21:54:36.672685 21000 raft_consensus.cc:2781] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 21:54:36.672784 21000 raft_consensus.cc:482] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 21:54:36.672850 21000 raft_consensus.cc:3037] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:36.673969 21000 raft_consensus.cc:504] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad5c2f0e234f90b031337e6d18cde0" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41643 } } peers { permanent_uuid: "c7918819891b4d61b00a08123b35898d" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38141 } } peers { permanent_uuid: "09d5905b9343414bb1598034b581dce2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45535 } }
I20260620 21:54:36.674347 21000 leader_election.cc:288] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [CANDIDATE]: Term 1 election: Requested vote from peers f1ad5c2f0e234f90b031337e6d18cde0 (127.0.193.194:41643), c7918819891b4d61b00a08123b35898d (127.0.193.193:38141)
I20260620 21:54:36.674975 20667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131b90b022434163b962b157e5a18a43" candidate_uuid: "09d5905b9343414bb1598034b581dce2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7918819891b4d61b00a08123b35898d"
I20260620 21:54:36.675146 20667 raft_consensus.cc:3037] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:36.676540 20667 raft_consensus.cc:2454] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d5905b9343414bb1598034b581dce2 in term 1.
I20260620 21:54:36.676898 20798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131b90b022434163b962b157e5a18a43" candidate_uuid: "09d5905b9343414bb1598034b581dce2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ad5c2f0e234f90b031337e6d18cde0"
I20260620 21:54:36.676993 20864 leader_election.cc:302] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09d5905b9343414bb1598034b581dce2, c7918819891b4d61b00a08123b35898d; no voters:
I20260620 21:54:36.677078 20798 raft_consensus.cc:3037] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:36.677196 21000 raft_consensus.cc:2781] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:36.677434 21000 raft_consensus.cc:686] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [term 1 LEADER]: Becoming Leader. State: Replica: 09d5905b9343414bb1598034b581dce2, State: Running, Role: LEADER
I20260620 21:54:36.677803 21000 consensus_queue.cc:237] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad5c2f0e234f90b031337e6d18cde0" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41643 } } peers { permanent_uuid: "c7918819891b4d61b00a08123b35898d" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38141 } } peers { permanent_uuid: "09d5905b9343414bb1598034b581dce2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45535 } }
I20260620 21:54:36.678270 20798 raft_consensus.cc:2454] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09d5905b9343414bb1598034b581dce2 in term 1.
I20260620 21:54:36.681174 20525 catalog_manager.cc:5495] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09d5905b9343414bb1598034b581dce2 (127.0.193.195). New cstate: current_term: 1 leader_uuid: "09d5905b9343414bb1598034b581dce2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad5c2f0e234f90b031337e6d18cde0" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7918819891b4d61b00a08123b35898d" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09d5905b9343414bb1598034b581dce2" member_type: VOTER last_known_addr { host: "127.0.193.195" port: 45535 } health_report { overall_health: HEALTHY } } }
I20260620 21:54:36.709403 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.705700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36262 (local address 127.0.193.195:45535)
0620 21:54:36.705861 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:36.705864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:36.705879 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:54:36.705957 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:36.705959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:36.706032 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:36.706126 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:54:36.706394 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.706962 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.707997 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:36.708214 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:36.708304 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:36.708596 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:54:36.709032 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:54:36.709124 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:36.709236 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 21:54:36.709286 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 21:54:36.716193 20667 raft_consensus.cc:1261] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d [term 1 FOLLOWER]: Refusing update from remote peer 09d5905b9343414bb1598034b581dce2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:54:36.716193 20798 raft_consensus.cc:1261] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0 [term 1 FOLLOWER]: Refusing update from remote peer 09d5905b9343414bb1598034b581dce2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:54:36.717060 21000 consensus_queue.cc:1035] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1ad5c2f0e234f90b031337e6d18cde0" member_type: VOTER last_known_addr { host: "127.0.193.194" port: 41643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:54:36.717374 21002 consensus_queue.cc:1035] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7918819891b4d61b00a08123b35898d" member_type: VOTER last_known_addr { host: "127.0.193.193" port: 38141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:54:36.725761 21005 mvcc.cc:205] Tried to move back new op lower bound from 7299041184620769280 to 7299041184475709440. Current Snapshot: MvccSnapshot[applied={T|T < 7299041184620769280}]
I20260620 21:54:36.728756 21007 mvcc.cc:205] Tried to move back new op lower bound from 7299041184620769280 to 7299041184475709440. Current Snapshot: MvccSnapshot[applied={T|T < 7299041184620769280 or (T in {7299041184620769280})}]
I20260620 21:54:36.738493 20525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:54:36.738714 20525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 21:54:36.741968 20525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5b77a9f129f46f280bf28ecc45e7dba: Sending DeleteTablet for 3 replicas of tablet 131b90b022434163b962b157e5a18a43
I20260620 21:54:36.743122 20778 tablet_service.cc:1514] Processing DeleteTablet for tablet 131b90b022434163b962b157e5a18a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:36 UTC) from {username='slave'} at 127.0.0.1:32950
I20260620 21:54:36.743396 20647 tablet_service.cc:1514] Processing DeleteTablet for tablet 131b90b022434163b962b157e5a18a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:36 UTC) from {username='slave'} at 127.0.0.1:51480
I20260620 21:54:36.743485 20908 tablet_service.cc:1514] Processing DeleteTablet for tablet 131b90b022434163b962b157e5a18a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 21:54:36 UTC) from {username='slave'} at 127.0.0.1:36252
I20260620 21:54:36.743939 21016 tablet_replica.cc:330] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d: stopping tablet replica
I20260620 21:54:36.744185 21015 tablet_replica.cc:330] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0: stopping tablet replica
I20260620 21:54:36.744329 21016 raft_consensus.cc:2229] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:54:36.744575 21017 tablet_replica.cc:330] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2: stopping tablet replica
I20260620 21:54:36.744719 21016 raft_consensus.cc:2258] T 131b90b022434163b962b157e5a18a43 P c7918819891b4d61b00a08123b35898d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:54:36.744755 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 20585
I20260620 21:54:36.744882 21017 raft_consensus.cc:2229] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 21:54:36.745285 21017 raft_consensus.cc:2258] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:54:36.746990 21017 ts_tablet_manager.cc:1905] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:54:36.748473 21015 raft_consensus.cc:2229] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 21:54:36.748889 21015 raft_consensus.cc:2258] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 21:54:36.750065 21017 ts_tablet_manager.cc:1918] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 21:54:36.750173 21017 log.cc:1196] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2: Deleting WAL directory at /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781992210835692-775-0/minicluster-data/ts-2/wal/wals/131b90b022434163b962b157e5a18a43
I20260620 21:54:36.750684 21017 ts_tablet_manager.cc:1939] T 131b90b022434163b962b157e5a18a43 P 09d5905b9343414bb1598034b581dce2: Deleting consensus metadata
I20260620 21:54:36.751473 21015 ts_tablet_manager.cc:1905] T 131b90b022434163b962b157e5a18a43 P f1ad5c2f0e234f90b031337e6d18cde0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 21:54:36.752238 20511 catalog_manager.cc:4841] TS 09d5905b9343414bb1598034b581dce2 (127.0.193.195:45535): tablet 131b90b022434163b962b157e5a18a43 (table test-table [id=fdebc216b50f4d829b9010323bfa5260]) successfully deleted
W20260620 21:54:36.753665 20511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.193.193:38141 (error 108)
I20260620 21:54:36.754768 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.754462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.193:38141 (local address 127.0.0.1:51490)
0620 21:54:36.754607 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:36.754681 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.193:38141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 21:54:36.754841 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 20716
W20260620 21:54:36.755517 20511 catalog_manager.cc:4584] TS c7918819891b4d61b00a08123b35898d (127.0.193.193:38141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 131b90b022434163b962b157e5a18a43: Network error: Client connection negotiation failed: client connection to 127.0.193.193:38141: connect: Connection refused (error 111)
I20260620 21:54:36.760684 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 20847
I20260620 21:54:36.762136 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:36.761736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.194:41643 (local address 127.0.0.1:32952)
0620 21:54:36.761964 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 21:54:36.762027 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.194:41643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 21:54:36.766697 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 20494
2026-06-20T21:54:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1170 ms)
[----------] 6 tests from EncryptionPolicyTest (11523 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:36 dist-test-slave-kk5c krb5kdc[21021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:36 dist-test-slave-kk5c krb5kdc[21021](info): set up 2 sockets
Jun 20 21:54:36 dist-test-slave-kk5c krb5kdc[21021](info): commencing operation
krb5kdc: starting...
W20260620 21:54:37.843488 775 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:37 dist-test-slave-kk5c krb5kdc[21021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:54:37.982892 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:38.109232 21034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:38.109700 21034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:38.109824 21034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:38.119083 21034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:38.119225 21034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:38.119261 21034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:38.119303 21034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:38.119344 21034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:38.126685 21034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:38.127935 21034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:38.128945 21034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:38.138263 21040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:38.138237 21039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:38.138237 21042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:38.138515 21034 server_base.cc:1034] running on GCE node
I20260620 21:54:38.139055 21034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:38.140347 21034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:38.141559 21034 hybrid_clock.cc:648] HybridClock initialized: now 1781992478141505 us; error 79 us; skew 500 ppm
Jun 20 21:54:38 dist-test-slave-kk5c krb5kdc[21021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:38.144654 21034 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:54:38.145570 21034 webserver.cc:460] Webserver started at http://127.0.0.1:46227/ using document root <none> and password file <none>
I20260620 21:54:38.146109 21034 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:38.146169 21034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:38.146340 21034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:38.148512 21034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "f2f17d8b02134530a2148c0dd562a8ce"
format_stamp: "Formatted at 2026-06-20 21:54:38 on dist-test-slave-kk5c"
server_key: "ab13b6e3dd34dcd5061350bb301b89b1"
server_key_iv: "6b564a723e30891890bde4eeaa0a6eca"
server_key_version: "encryptionkey@0"
I20260620 21:54:38.149058 21034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "f2f17d8b02134530a2148c0dd562a8ce"
format_stamp: "Formatted at 2026-06-20 21:54:38 on dist-test-slave-kk5c"
server_key: "ab13b6e3dd34dcd5061350bb301b89b1"
server_key_iv: "6b564a723e30891890bde4eeaa0a6eca"
server_key_version: "encryptionkey@0"
I20260620 21:54:38.153110 21034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:54:38.156347 21049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:38.157788 21034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:38.157997 21034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "f2f17d8b02134530a2148c0dd562a8ce"
format_stamp: "Formatted at 2026-06-20 21:54:38 on dist-test-slave-kk5c"
server_key: "ab13b6e3dd34dcd5061350bb301b89b1"
server_key_iv: "6b564a723e30891890bde4eeaa0a6eca"
server_key_version: "encryptionkey@0"
I20260620 21:54:38.158145 21034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:38.175772 21034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:38.179620 21034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:38.179839 21034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:38.187490 21034 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36893
I20260620 21:54:38.187511 21101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36893 every 8 connection(s)
I20260620 21:54:38.188546 21034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:38.190364 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 21034
I20260620 21:54:38.190516 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:38.190805 775 external_mini_cluster.cc:1428] Setting key 81399cc9f71ef6ff2c397a911a31a39b
I20260620 21:54:38.191733 21102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:38.198704 21102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce: Bootstrap starting.
Jun 20 21:54:38 dist-test-slave-kk5c krb5kdc[21021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 21:54:38.201380 21102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:38.202380 21102 log.cc:824] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:38.204764 21102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce: No bootstrap required, opened a new log
I20260620 21:54:38.206456 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:38.192287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44339 (local address 127.0.0.1:36893)
0620 21:54:38.192701 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:38.192719 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:38.192770 (+ 51us) server_negotiation.cc:407] Connection header received
0620 21:54:38.193395 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:38.193410 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:38.193781 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:38.194061 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:38.195025 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:38.196109 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:38.197338 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:38.197884 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:38.198013 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:38.200967 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:38.200973 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:38.200987 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:38.201016 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:38.203023 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:38.203593 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:38.203599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:38.203605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:38.203738 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:38.204086 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:38.204089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:38.204091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:38.204603 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:38.204825 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:38.205230 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 21:54:38.205478 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":151,"threads_started":1}
I20260620 21:54:38.209326 21102 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f17d8b02134530a2148c0dd562a8ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36893 } }
I20260620 21:54:38.209599 21102 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:38.209638 21102 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2f17d8b02134530a2148c0dd562a8ce, State: Initialized, Role: FOLLOWER
I20260620 21:54:38.210120 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:38.210139 21102 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f17d8b02134530a2148c0dd562a8ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36893 } }
I20260620 21:54:38.210309 21102 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:38.210381 21102 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:38.210520 21102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:38.211699 21102 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f17d8b02134530a2148c0dd562a8ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36893 } }
I20260620 21:54:38.212101 21102 leader_election.cc:302] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2f17d8b02134530a2148c0dd562a8ce; no voters:
I20260620 21:54:38.212445 21102 leader_election.cc:288] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:38.212692 21111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:38.212966 21111 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [term 1 LEADER]: Becoming Leader. State: Replica: f2f17d8b02134530a2148c0dd562a8ce, State: Running, Role: LEADER
I20260620 21:54:38.213299 21111 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f17d8b02134530a2148c0dd562a8ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36893 } }
I20260620 21:54:38.213902 21102 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:38.214634 21111 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2f17d8b02134530a2148c0dd562a8ce. Latest consensus state: current_term: 1 leader_uuid: "f2f17d8b02134530a2148c0dd562a8ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f17d8b02134530a2148c0dd562a8ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36893 } } }
I20260620 21:54:38.214799 21111 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:38.215077 21114 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2f17d8b02134530a2148c0dd562a8ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f17d8b02134530a2148c0dd562a8ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36893 } } }
I20260620 21:54:38.215190 21114 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce [sys.catalog]: This master's current role is: LEADER
W20260620 21:54:38.219647 21127 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:54:38.219786 21127 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:54:38.220103 21121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:38.221596 21121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:38.227988 21121 catalog_manager.cc:1347] Generated new cluster ID: 31f4204e5be64b19b308f1c944715e84
I20260620 21:54:38.228152 21121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:38.228281 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:38.212712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41469 (local address 127.0.0.1:36893)
0620 21:54:38.212838 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:38.212842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:38.216560 (+ 3718us) server_negotiation.cc:407] Connection header received
0620 21:54:38.216928 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:38.216932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:38.216983 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:38.217087 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:38.217812 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:38.218572 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:38.219777 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:38.220338 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:38.220689 (+ 351us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:38.223014 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:38.223020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:38.223024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:38.223052 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:38.225169 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:38.225959 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:38.225964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:38.225966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:38.226070 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:38.226584 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:38.226588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:38.226590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:38.226865 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:38.227009 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:38.227903 (+ 894us) server_negotiation.cc:299] Negotiation successful
0620 21:54:38.228040 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 21:54:38.237061 21121 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:38.238081 21121 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:38.245632 21121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2f17d8b02134530a2148c0dd562a8ce: Generated new TSK 0
I20260620 21:54:38.246495 21121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:38.255038 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 21034
W20260620 21:54:38.260524 21019 connection.cc:535] client connection to 127.0.0.1:36893 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36893): Connection reset by peer (error 104)
2026-06-20T21:54:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:38 dist-test-slave-kk5c krb5kdc[21133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:38 dist-test-slave-kk5c krb5kdc[21133](info): set up 2 sockets
Jun 20 21:54:38 dist-test-slave-kk5c krb5kdc[21133](info): commencing operation
krb5kdc: starting...
W20260620 21:54:39.347559 775 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:39 dist-test-slave-kk5c krb5kdc[21133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:54:39.482491 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:39.597196 21146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:39.597569 21146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:39.597663 21146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:39.601935 21146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:39.602059 21146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:39.602095 21146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:39.602169 21146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:39.602226 21146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:39.608151 21146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:39.609385 21146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:39.610460 21146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:39.621737 21152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:39.621762 21151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:39.622359 21154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:39.624758 21146 server_base.cc:1034] running on GCE node
I20260620 21:54:39.625597 21146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:39.627064 21146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:39.628295 21146 hybrid_clock.cc:648] HybridClock initialized: now 1781992479628264 us; error 49 us; skew 500 ppm
Jun 20 21:54:39 dist-test-slave-kk5c krb5kdc[21133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:39.632265 21146 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:54:39.633266 21146 webserver.cc:460] Webserver started at http://127.0.0.1:40731/ using document root <none> and password file <none>
I20260620 21:54:39.633873 21146 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:39.633937 21146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:39.634121 21146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:39.636363 21146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "1720336f71d946668f966da3c259ae5e"
format_stamp: "Formatted at 2026-06-20 21:54:39 on dist-test-slave-kk5c"
server_key: "7d949b4501bdd5fe4a1d8f7a4c548848"
server_key_iv: "95c52159ff8e07fc6a21d1a6cd5a80d8"
server_key_version: "encryptionkey@0"
I20260620 21:54:39.636936 21146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "1720336f71d946668f966da3c259ae5e"
format_stamp: "Formatted at 2026-06-20 21:54:39 on dist-test-slave-kk5c"
server_key: "7d949b4501bdd5fe4a1d8f7a4c548848"
server_key_iv: "95c52159ff8e07fc6a21d1a6cd5a80d8"
server_key_version: "encryptionkey@0"
I20260620 21:54:39.641294 21146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 21:54:39.644487 21161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:39.646044 21146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:54:39.646185 21146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "1720336f71d946668f966da3c259ae5e"
format_stamp: "Formatted at 2026-06-20 21:54:39 on dist-test-slave-kk5c"
server_key: "7d949b4501bdd5fe4a1d8f7a4c548848"
server_key_iv: "95c52159ff8e07fc6a21d1a6cd5a80d8"
server_key_version: "encryptionkey@0"
I20260620 21:54:39.646430 21146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:39.657253 21146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:39.660876 21146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:39.661116 21146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:39.669202 21146 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37261
I20260620 21:54:39.669237 21213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37261 every 8 connection(s)
I20260620 21:54:39.670276 21146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:39.673295 21214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:39.679457 21214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e: Bootstrap starting.
I20260620 21:54:39.679971 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 21146
I20260620 21:54:39.680150 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:39.680433 775 external_mini_cluster.cc:1428] Setting key 57beb16f2b97ffd46037a550667ea262
I20260620 21:54:39.682529 21214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:39.683389 21214 log.cc:824] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:39.685619 21214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e: No bootstrap required, opened a new log
I20260620 21:54:39.689913 21214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1720336f71d946668f966da3c259ae5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260620 21:54:39.690203 21214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:39.690292 21214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1720336f71d946668f966da3c259ae5e, State: Initialized, Role: FOLLOWER
I20260620 21:54:39.690331 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:39.681917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41231 (local address 127.0.0.1:37261)
0620 21:54:39.682567 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:39.682583 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:39.682618 (+ 35us) server_negotiation.cc:407] Connection header received
0620 21:54:39.683446 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:39.683467 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:39.683915 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:39.684269 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:39.684753 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:39.685686 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:39.686649 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:39.687160 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:39.687414 (+ 254us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:39.688457 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:39.688462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:39.688477 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:39.688896 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:39.688953 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:39.688976 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:39.689124 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:39.689334 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 21:54:39.689484 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":172,"threads_started":1}
I20260620 21:54:39.690857 21214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1720336f71d946668f966da3c259ae5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260620 21:54:39.691048 21214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:39.691118 21214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:39.691241 21214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:39.692390 21214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1720336f71d946668f966da3c259ae5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260620 21:54:39.692760 21214 leader_election.cc:302] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1720336f71d946668f966da3c259ae5e; no voters:
I20260620 21:54:39.693167 21214 leader_election.cc:288] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:39.693243 21219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:39.693485 21219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [term 1 LEADER]: Becoming Leader. State: Replica: 1720336f71d946668f966da3c259ae5e, State: Running, Role: LEADER
I20260620 21:54:39.693646 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:39.693825 21219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1720336f71d946668f966da3c259ae5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260620 21:54:39.694628 21214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:39.695233 21219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1720336f71d946668f966da3c259ae5e. Latest consensus state: current_term: 1 leader_uuid: "1720336f71d946668f966da3c259ae5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1720336f71d946668f966da3c259ae5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } } }
I20260620 21:54:39.695364 21219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:39.695602 21219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1720336f71d946668f966da3c259ae5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1720336f71d946668f966da3c259ae5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } } }
I20260620 21:54:39.695710 21219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:39.696192 21228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:39.699807 21228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:39.701162 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:39.696077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60625 (local address 127.0.0.1:37261)
0620 21:54:39.696180 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:39.696185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:39.696249 (+ 64us) server_negotiation.cc:407] Connection header received
0620 21:54:39.696457 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:39.696460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:39.696516 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:39.696569 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:39.696996 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:39.697755 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:39.698983 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:39.699387 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:39.699485 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:39.700646 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:39.700649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:39.700652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:39.700665 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:39.700699 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:39.700710 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:39.700804 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:39.700965 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 21:54:39.701000 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 21:54:39.706182 21228 catalog_manager.cc:1347] Generated new cluster ID: fbfa767d5261433db39a521fa15af1ae
I20260620 21:54:39.706265 21228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:39.715431 21228 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:39.716373 21228 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:39.720935 21228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1720336f71d946668f966da3c259ae5e: Generated new TSK 0
I20260620 21:54:39.721709 21228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:39.726734 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 21146
W20260620 21:54:39.731932 21131 connection.cc:535] client connection to 127.0.0.1:37261 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37261): Connection reset by peer (error 104)
2026-06-20T21:54:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:39 dist-test-slave-kk5c krb5kdc[21245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:39 dist-test-slave-kk5c krb5kdc[21245](info): set up 2 sockets
Jun 20 21:54:39 dist-test-slave-kk5c krb5kdc[21245](info): commencing operation
krb5kdc: starting...
W20260620 21:54:40.784034 775 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:40 dist-test-slave-kk5c krb5kdc[21245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:54:40.916457 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:41.038998 21258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:41.039407 21258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:41.039497 21258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:41.043602 21258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:41.043711 21258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:41.043751 21258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:41.043792 21258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:41.043824 21258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:41.049329 21258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41937
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:41.050472 21258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:41.051530 21258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:41.060382 21263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:41.060401 21264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:41.060458 21258 server_base.cc:1034] running on GCE node
W20260620 21:54:41.060372 21266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:41.061357 21258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:41.062623 21258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:41.063851 21258 hybrid_clock.cc:648] HybridClock initialized: now 1781992481063831 us; error 35 us; skew 500 ppm
Jun 20 21:54:41 dist-test-slave-kk5c krb5kdc[21245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:41.071743 21258 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:54:41.072798 21258 webserver.cc:460] Webserver started at http://127.0.0.1:41923/ using document root <none> and password file <none>
I20260620 21:54:41.073554 21258 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:41.073623 21258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:41.073894 21258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:41.076262 21258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "49d40bdeaca3452f83370f010a33a37f"
format_stamp: "Formatted at 2026-06-20 21:54:41 on dist-test-slave-kk5c"
server_key: "b3a0e57b497549eb217ad6b5f117c303"
server_key_iv: "ad536b7e5ecea2f197305bc1a298e674"
server_key_version: "encryptionkey@0"
I20260620 21:54:41.076929 21258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "49d40bdeaca3452f83370f010a33a37f"
format_stamp: "Formatted at 2026-06-20 21:54:41 on dist-test-slave-kk5c"
server_key: "b3a0e57b497549eb217ad6b5f117c303"
server_key_iv: "ad536b7e5ecea2f197305bc1a298e674"
server_key_version: "encryptionkey@0"
I20260620 21:54:41.081727 21258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:54:41.084491 21273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:41.085952 21258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 21:54:41.086138 21258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "49d40bdeaca3452f83370f010a33a37f"
format_stamp: "Formatted at 2026-06-20 21:54:41 on dist-test-slave-kk5c"
server_key: "b3a0e57b497549eb217ad6b5f117c303"
server_key_iv: "ad536b7e5ecea2f197305bc1a298e674"
server_key_version: "encryptionkey@0"
I20260620 21:54:41.086279 21258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:41.092051 21258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:41.095597 21258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:41.095919 21258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:41.103910 21258 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41937
I20260620 21:54:41.103976 21325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41937 every 8 connection(s)
I20260620 21:54:41.104909 21258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:41.107892 21326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:41.113813 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 21258
I20260620 21:54:41.113955 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:41.113927 21326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f: Bootstrap starting.
I20260620 21:54:41.114295 775 external_mini_cluster.cc:1428] Setting key 998acf51635f63c10b50fc9fdb3de929
I20260620 21:54:41.117075 21326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:41.117950 21326 log.cc:824] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:41.120609 21326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f: No bootstrap required, opened a new log
I20260620 21:54:41.120802 21329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:41.115846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34325 (local address 127.0.0.1:41937)
0620 21:54:41.116541 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:41.116565 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:41.116606 (+ 41us) server_negotiation.cc:407] Connection header received
0620 21:54:41.117664 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:41.117687 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:41.118060 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:41.118627 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:41.119449 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:41.119455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:41.119468 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:41.119483 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:41.119543 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:41.119563 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:41.119724 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:41.119831 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 21:54:41.120004 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":146,"threads_started":1}
I20260620 21:54:41.124692 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:41.125649 21326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d40bdeaca3452f83370f010a33a37f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } }
I20260620 21:54:41.125960 21326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:41.126052 21326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d40bdeaca3452f83370f010a33a37f, State: Initialized, Role: FOLLOWER
I20260620 21:54:41.126576 21326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d40bdeaca3452f83370f010a33a37f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } }
I20260620 21:54:41.126749 21326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:41.126827 21326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:41.126958 21326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:41.128356 21326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d40bdeaca3452f83370f010a33a37f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } }
I20260620 21:54:41.128753 21326 leader_election.cc:302] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49d40bdeaca3452f83370f010a33a37f; no voters:
I20260620 21:54:41.129101 21326 leader_election.cc:288] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:41.129230 21336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:41.129446 21336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [term 1 LEADER]: Becoming Leader. State: Replica: 49d40bdeaca3452f83370f010a33a37f, State: Running, Role: LEADER
I20260620 21:54:41.129482 21329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:41.127589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48485 (local address 127.0.0.1:41937)
0620 21:54:41.127737 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:41.127743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:41.127957 (+ 214us) server_negotiation.cc:407] Connection header received
0620 21:54:41.128138 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:41.128141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:41.128179 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:41.128282 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:41.129037 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:41.129041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:41.129044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:41.129055 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:41.129082 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:41.129100 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:41.129193 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:41.129283 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 21:54:41.129326 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 21:54:41.130008 21336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d40bdeaca3452f83370f010a33a37f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } }
I20260620 21:54:41.130585 21326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:41.131966 21338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49d40bdeaca3452f83370f010a33a37f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d40bdeaca3452f83370f010a33a37f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } } }
I20260620 21:54:41.131994 21337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49d40bdeaca3452f83370f010a33a37f. Latest consensus state: current_term: 1 leader_uuid: "49d40bdeaca3452f83370f010a33a37f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d40bdeaca3452f83370f010a33a37f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } } }
I20260620 21:54:41.132118 21338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:41.132118 21337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:41.132568 21346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:41.135474 21346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:41.141727 21346 catalog_manager.cc:1347] Generated new cluster ID: 54b6ed29cb08423b92c49b7391a3f966
I20260620 21:54:41.141824 21346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:41.155419 21346 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:41.155943 21346 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:41.172492 21346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49d40bdeaca3452f83370f010a33a37f: Generated new TSK 0
I20260620 21:54:41.173344 21346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:41.187127 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 21258
2026-06-20T21:54:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:41 dist-test-slave-kk5c krb5kdc[21357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:41 dist-test-slave-kk5c krb5kdc[21357](info): set up 2 sockets
Jun 20 21:54:41 dist-test-slave-kk5c krb5kdc[21357](info): commencing operation
krb5kdc: starting...
W20260620 21:54:42.259586 775 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:42 dist-test-slave-kk5c krb5kdc[21357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:54:42.390937 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46021
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:42.510380 21370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:42.510679 21370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:42.510725 21370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:42.514784 21370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:42.514863 21370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:42.514883 21370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:42.514901 21370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:42.514917 21370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:42.520510 21370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:42.521607 21370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:42.522651 21370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:42.531692 21378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:42.531692 21375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:42.531694 21376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:42.531986 21370 server_base.cc:1034] running on GCE node
I20260620 21:54:42.532866 21370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:42.534152 21370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:42.535280 21370 hybrid_clock.cc:648] HybridClock initialized: now 1781992482535267 us; error 31 us; skew 500 ppm
Jun 20 21:54:42 dist-test-slave-kk5c krb5kdc[21357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:42.538470 21370 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:54:42.539565 21370 webserver.cc:460] Webserver started at http://127.0.0.1:40989/ using document root <none> and password file <none>
I20260620 21:54:42.540150 21370 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:42.540244 21370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:42.540485 21370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:42.542791 21370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "3129a7d44c764bab8a455c7ff95d6c6f"
format_stamp: "Formatted at 2026-06-20 21:54:42 on dist-test-slave-kk5c"
server_key: "1be8d5ce2e336aefe0a55c204fedec05"
server_key_iv: "69b84ff5d63e308bc7434a50e4cc5003"
server_key_version: "encryptionkey@0"
I20260620 21:54:42.543481 21370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "3129a7d44c764bab8a455c7ff95d6c6f"
format_stamp: "Formatted at 2026-06-20 21:54:42 on dist-test-slave-kk5c"
server_key: "1be8d5ce2e336aefe0a55c204fedec05"
server_key_iv: "69b84ff5d63e308bc7434a50e4cc5003"
server_key_version: "encryptionkey@0"
I20260620 21:54:42.547533 21370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:54:42.550108 21385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:42.551726 21370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:42.551890 21370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "3129a7d44c764bab8a455c7ff95d6c6f"
format_stamp: "Formatted at 2026-06-20 21:54:42 on dist-test-slave-kk5c"
server_key: "1be8d5ce2e336aefe0a55c204fedec05"
server_key_iv: "69b84ff5d63e308bc7434a50e4cc5003"
server_key_version: "encryptionkey@0"
I20260620 21:54:42.552038 21370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:42.586809 21370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:42.590430 21370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:42.590662 21370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:42.598080 21370 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46021
I20260620 21:54:42.598125 21437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46021 every 8 connection(s)
I20260620 21:54:42.599157 21370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:42.602519 21438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:42.608716 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 21370
I20260620 21:54:42.608901 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:42.609263 775 external_mini_cluster.cc:1428] Setting key 31c2ffe4041940c5ca8f760a65c7c62f
I20260620 21:54:42.610087 21438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f: Bootstrap starting.
I20260620 21:54:42.612871 21438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:42.613734 21438 log.cc:824] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:42.616034 21438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f: No bootstrap required, opened a new log
Jun 20 21:54:42 dist-test-slave-kk5c krb5kdc[21357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 21:54:42.621258 21438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3129a7d44c764bab8a455c7ff95d6c6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46021 } }
I20260620 21:54:42.621632 21438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:42.621706 21438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3129a7d44c764bab8a455c7ff95d6c6f, State: Initialized, Role: FOLLOWER
I20260620 21:54:42.622327 21438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3129a7d44c764bab8a455c7ff95d6c6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46021 } }
I20260620 21:54:42.622515 21438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:42.622620 21438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:42.622740 21438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:42.624120 21438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3129a7d44c764bab8a455c7ff95d6c6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46021 } }
I20260620 21:54:42.624589 21438 leader_election.cc:302] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3129a7d44c764bab8a455c7ff95d6c6f; no voters:
I20260620 21:54:42.624984 21438 leader_election.cc:288] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:42.625126 21443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:42.625422 21443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [term 1 LEADER]: Becoming Leader. State: Replica: 3129a7d44c764bab8a455c7ff95d6c6f, State: Running, Role: LEADER
I20260620 21:54:42.625792 21443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3129a7d44c764bab8a455c7ff95d6c6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46021 } }
I20260620 21:54:42.626492 21438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:42.627097 21443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3129a7d44c764bab8a455c7ff95d6c6f. Latest consensus state: current_term: 1 leader_uuid: "3129a7d44c764bab8a455c7ff95d6c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3129a7d44c764bab8a455c7ff95d6c6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46021 } } }
I20260620 21:54:42.627264 21443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:42.627601 21445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3129a7d44c764bab8a455c7ff95d6c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3129a7d44c764bab8a455c7ff95d6c6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46021 } } }
I20260620 21:54:42.627694 21445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:42.627744 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:42.610956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53545 (local address 127.0.0.1:46021)
0620 21:54:42.611501 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:42.611523 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:42.611563 (+ 40us) server_negotiation.cc:407] Connection header received
0620 21:54:42.612311 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:42.612330 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:42.612716 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:42.613118 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:42.614094 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:42.615260 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:42.616293 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:42.617055 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:42.617263 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:42.620752 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:42.620760 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:42.620782 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:42.620824 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:42.623314 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:42.623866 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:42.623876 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:42.623886 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:42.624063 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:42.624509 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:42.624512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:42.624514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:42.625152 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:42.625368 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:42.626219 (+ 851us) server_negotiation.cc:299] Negotiation successful
0620 21:54:42.626565 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":182,"threads_started":1}
I20260620 21:54:42.627812 21448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:42.631388 21448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:42.631614 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:42.638338 21448 catalog_manager.cc:1347] Generated new cluster ID: 47f9609c484e4b98a21d96d40da39db6
I20260620 21:54:42.638437 21448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:42.644285 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:42.634097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34025 (local address 127.0.0.1:46021)
0620 21:54:42.634248 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:42.634253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:42.634524 (+ 271us) server_negotiation.cc:407] Connection header received
0620 21:54:42.634668 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:42.634671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:42.634727 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:42.634824 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:42.635831 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:42.636512 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:42.637459 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:42.637754 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:42.637878 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:42.639982 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:42.639986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:42.639989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:42.640020 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:42.642101 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:42.642784 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:42.642788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:42.642838 (+ 50us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:42.642937 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:42.643268 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:42.643271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:42.643273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:42.643495 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:42.643600 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:42.643970 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 21:54:42.644105 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 21:54:42.646541 21448 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:42.647835 21448 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:42.656749 21448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3129a7d44c764bab8a455c7ff95d6c6f: Generated new TSK 0
I20260620 21:54:42.657527 21448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:42.664449 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 21370
2026-06-20T21:54:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:42 dist-test-slave-kk5c krb5kdc[21469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:42 dist-test-slave-kk5c krb5kdc[21469](info): set up 2 sockets
Jun 20 21:54:42 dist-test-slave-kk5c krb5kdc[21469](info): commencing operation
krb5kdc: starting...
W20260620 21:54:43.719545 775 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:43 dist-test-slave-kk5c krb5kdc[21469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:54:43.851330 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:43.969590 21482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:43.969949 21482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:43.970033 21482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:43.974321 21482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:43.974431 21482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:43.974468 21482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:43.974529 21482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:43.974566 21482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:43.980258 21482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:43.981436 21482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:43.982472 21482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:43.991398 21487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:43.991398 21488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:43.991526 21482 server_base.cc:1034] running on GCE node
W20260620 21:54:43.991437 21490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:43.992362 21482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:43.993839 21482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:43.995020 21482 hybrid_clock.cc:648] HybridClock initialized: now 1781992483994993 us; error 39 us; skew 500 ppm
Jun 20 21:54:43 dist-test-slave-kk5c krb5kdc[21469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:43.998083 21482 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:54:43.999086 21482 webserver.cc:460] Webserver started at http://127.0.0.1:44523/ using document root <none> and password file <none>
I20260620 21:54:43.999773 21482 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:43.999874 21482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:44.000115 21482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:44.002238 21482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "5209ae491467482185219a6968cb119f"
format_stamp: "Formatted at 2026-06-20 21:54:43 on dist-test-slave-kk5c"
server_key: "f4075549145e45585f2eb94c0fdc3897"
server_key_iv: "04d5b6e7a4105ddb11d438fb40549ec9"
server_key_version: "encryptionkey@0"
I20260620 21:54:44.002844 21482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "5209ae491467482185219a6968cb119f"
format_stamp: "Formatted at 2026-06-20 21:54:43 on dist-test-slave-kk5c"
server_key: "f4075549145e45585f2eb94c0fdc3897"
server_key_iv: "04d5b6e7a4105ddb11d438fb40549ec9"
server_key_version: "encryptionkey@0"
I20260620 21:54:44.006798 21482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:54:44.009382 21497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:44.010715 21482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 21:54:44.010885 21482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "5209ae491467482185219a6968cb119f"
format_stamp: "Formatted at 2026-06-20 21:54:43 on dist-test-slave-kk5c"
server_key: "f4075549145e45585f2eb94c0fdc3897"
server_key_iv: "04d5b6e7a4105ddb11d438fb40549ec9"
server_key_version: "encryptionkey@0"
I20260620 21:54:44.011019 21482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:44.028682 21482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:44.031948 21482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:44.032173 21482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:44.039932 21482 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41233
I20260620 21:54:44.039958 21549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41233 every 8 connection(s)
I20260620 21:54:44.040970 21482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:44.044122 21550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:44.048727 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 21482
I20260620 21:54:44.048856 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:44.049185 775 external_mini_cluster.cc:1428] Setting key de2d7f633e746f727504936625f612bd
I20260620 21:54:44.050369 21550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f: Bootstrap starting.
I20260620 21:54:44.052795 21550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:44.053635 21550 log.cc:824] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:44.055703 21550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f: No bootstrap required, opened a new log
I20260620 21:54:44.058471 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:44.050700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44105 (local address 127.0.0.1:41233)
0620 21:54:44.051255 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:44.051281 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:44.051318 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:54:44.051990 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:44.052009 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:44.052368 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:44.052716 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:44.053165 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:44.054253 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:44.055236 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:44.055705 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:44.055891 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:44.056760 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:44.056765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:44.056779 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:44.057183 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:44.057240 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:44.057263 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:44.057378 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:44.057488 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 21:54:44.057714 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":141,"threads_started":1}
I20260620 21:54:44.059872 21550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209ae491467482185219a6968cb119f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41233 } }
I20260620 21:54:44.060163 21550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:44.060223 21550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5209ae491467482185219a6968cb119f, State: Initialized, Role: FOLLOWER
I20260620 21:54:44.060742 21550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209ae491467482185219a6968cb119f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41233 } }
I20260620 21:54:44.060886 21550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:44.061014 21550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:44.061142 21550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:44.061767 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:44.062311 21550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209ae491467482185219a6968cb119f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41233 } }
I20260620 21:54:44.062670 21550 leader_election.cc:302] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5209ae491467482185219a6968cb119f; no voters:
I20260620 21:54:44.063015 21550 leader_election.cc:288] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:44.063088 21556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:44.063292 21556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [term 1 LEADER]: Becoming Leader. State: Replica: 5209ae491467482185219a6968cb119f, State: Running, Role: LEADER
I20260620 21:54:44.063606 21556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209ae491467482185219a6968cb119f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41233 } }
I20260620 21:54:44.064379 21550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:44.064779 21556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5209ae491467482185219a6968cb119f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209ae491467482185219a6968cb119f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41233 } } }
I20260620 21:54:44.064889 21556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:44.065191 21560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5209ae491467482185219a6968cb119f. Latest consensus state: current_term: 1 leader_uuid: "5209ae491467482185219a6968cb119f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5209ae491467482185219a6968cb119f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41233 } } }
I20260620 21:54:44.065295 21560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:44.066861 21564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:44.069299 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:44.064024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42439 (local address 127.0.0.1:41233)
0620 21:54:44.064202 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:44.064206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:44.064419 (+ 213us) server_negotiation.cc:407] Connection header received
0620 21:54:44.064634 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:44.064640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:44.064694 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:44.064766 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:44.065258 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:44.066011 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:44.067339 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:44.067581 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:44.067712 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:44.068819 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:44.068823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:44.068826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:44.068840 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:44.068870 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:44.068883 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:44.068969 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:44.069097 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 21:54:44.069157 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 21:54:44.070276 21564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:44.076476 21564 catalog_manager.cc:1347] Generated new cluster ID: 2b72e2212e534c2f818592a59fd8c69f
I20260620 21:54:44.076567 21564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:44.108878 21564 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:44.110009 21564 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:44.121202 21564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5209ae491467482185219a6968cb119f: Generated new TSK 0
I20260620 21:54:44.122118 21564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:44.133440 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 21482
2026-06-20T21:54:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:44 dist-test-slave-kk5c krb5kdc[21581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:44 dist-test-slave-kk5c krb5kdc[21581](info): set up 2 sockets
Jun 20 21:54:44 dist-test-slave-kk5c krb5kdc[21581](info): commencing operation
krb5kdc: starting...
W20260620 21:54:45.199677 775 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:45 dist-test-slave-kk5c krb5kdc[21581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:54:45.331405 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:45.450171 21594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:45.450541 21594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:45.450585 21594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:45.454859 21594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:45.454936 21594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:45.454957 21594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:45.454975 21594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:45.454991 21594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:45.460893 21594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:45.462033 21594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:45.463121 21594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:45.472241 21600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:45.472244 21602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:45.472200 21599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:45.472446 21594 server_base.cc:1034] running on GCE node
I20260620 21:54:45.473202 21594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:45.474524 21594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:45.475745 21594 hybrid_clock.cc:648] HybridClock initialized: now 1781992485475729 us; error 32 us; skew 500 ppm
Jun 20 21:54:45 dist-test-slave-kk5c krb5kdc[21581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:45.478993 21594 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:54:45.480154 21594 webserver.cc:460] Webserver started at http://127.0.0.1:43601/ using document root <none> and password file <none>
I20260620 21:54:45.480815 21594 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:45.480919 21594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:45.481191 21594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:45.484189 21594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "b53c7416d1fe4acb98dd40a2ad4066e9"
format_stamp: "Formatted at 2026-06-20 21:54:45 on dist-test-slave-kk5c"
server_key: "c2f8bf946b8f533909e04fcf776668ea"
server_key_iv: "09e218369045137ca137f233be325e49"
server_key_version: "encryptionkey@0"
I20260620 21:54:45.484833 21594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "b53c7416d1fe4acb98dd40a2ad4066e9"
format_stamp: "Formatted at 2026-06-20 21:54:45 on dist-test-slave-kk5c"
server_key: "c2f8bf946b8f533909e04fcf776668ea"
server_key_iv: "09e218369045137ca137f233be325e49"
server_key_version: "encryptionkey@0"
I20260620 21:54:45.488902 21594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:54:45.491714 21609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:45.493127 21594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:45.493330 21594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "b53c7416d1fe4acb98dd40a2ad4066e9"
format_stamp: "Formatted at 2026-06-20 21:54:45 on dist-test-slave-kk5c"
server_key: "c2f8bf946b8f533909e04fcf776668ea"
server_key_iv: "09e218369045137ca137f233be325e49"
server_key_version: "encryptionkey@0"
I20260620 21:54:45.493467 21594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:45.498392 21594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:45.502167 21594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:45.502386 21594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:45.509971 21594 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41181
I20260620 21:54:45.509989 21661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41181 every 8 connection(s)
I20260620 21:54:45.510965 21594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:45.513827 21662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:45.518554 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 21594
I20260620 21:54:45.518720 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:45.519011 775 external_mini_cluster.cc:1428] Setting key e8d295be41a5791323ca65e55d4c42c0
I20260620 21:54:45.519822 21662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9: Bootstrap starting.
I20260620 21:54:45.522361 21662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:45.523409 21662 log.cc:824] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:45.524461 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:45.520421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46345 (local address 127.0.0.1:41181)
0620 21:54:45.520869 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:45.520884 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:45.520921 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:54:45.521567 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:45.521583 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:45.521877 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:45.522174 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:45.523088 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:45.523093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:45.523105 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:45.523116 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:45.523175 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:45.523282 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:45.523387 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:45.523556 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 21:54:45.523746 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":153,"threads_started":1}
I20260620 21:54:45.525451 21662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9: No bootstrap required, opened a new log
I20260620 21:54:45.527436 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:45.529525 21662 raft_consensus.cc:348] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53c7416d1fe4acb98dd40a2ad4066e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } }
I20260620 21:54:45.529850 21662 raft_consensus.cc:374] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:45.529930 21662 raft_consensus.cc:729] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53c7416d1fe4acb98dd40a2ad4066e9, State: Initialized, Role: FOLLOWER
I20260620 21:54:45.530419 21662 consensus_queue.cc:260] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53c7416d1fe4acb98dd40a2ad4066e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } }
I20260620 21:54:45.530565 21662 raft_consensus.cc:388] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:45.530635 21662 raft_consensus.cc:482] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:45.530758 21662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:45.531664 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:45.529617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45503 (local address 127.0.0.1:41181)
0620 21:54:45.529761 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:45.529765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:45.530028 (+ 263us) server_negotiation.cc:407] Connection header received
0620 21:54:45.530201 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:45.530206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:45.530285 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:45.530385 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:45.531166 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:45.531170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:45.531173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:45.531214 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:45.531243 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:45.531293 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:45.531399 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:45.531480 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 21:54:45.531527 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:54:45.532086 21662 raft_consensus.cc:504] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53c7416d1fe4acb98dd40a2ad4066e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } }
I20260620 21:54:45.532437 21662 leader_election.cc:302] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b53c7416d1fe4acb98dd40a2ad4066e9; no voters:
I20260620 21:54:45.532833 21662 leader_election.cc:288] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:45.533011 21672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:45.533279 21672 raft_consensus.cc:686] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [term 1 LEADER]: Becoming Leader. State: Replica: b53c7416d1fe4acb98dd40a2ad4066e9, State: Running, Role: LEADER
I20260620 21:54:45.533640 21672 consensus_queue.cc:237] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53c7416d1fe4acb98dd40a2ad4066e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } }
I20260620 21:54:45.534207 21662 sys_catalog.cc:564] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:45.535439 21673 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b53c7416d1fe4acb98dd40a2ad4066e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53c7416d1fe4acb98dd40a2ad4066e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } } }
I20260620 21:54:45.535584 21673 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:45.535918 21674 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b53c7416d1fe4acb98dd40a2ad4066e9. Latest consensus state: current_term: 1 leader_uuid: "b53c7416d1fe4acb98dd40a2ad4066e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53c7416d1fe4acb98dd40a2ad4066e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } } }
I20260620 21:54:45.536046 21674 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:45.536135 21681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:45.539861 21681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:45.546196 21681 catalog_manager.cc:1347] Generated new cluster ID: 134143ef8502464a83501c50001298c5
I20260620 21:54:45.546289 21681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:45.559170 21681 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:45.559603 21681 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:45.571141 21681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b53c7416d1fe4acb98dd40a2ad4066e9: Generated new TSK 0
I20260620 21:54:45.572014 21681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:45.581743 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 21594
2026-06-20T21:54:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:45 dist-test-slave-kk5c krb5kdc[21693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:45 dist-test-slave-kk5c krb5kdc[21693](info): set up 2 sockets
Jun 20 21:54:45 dist-test-slave-kk5c krb5kdc[21693](info): commencing operation
krb5kdc: starting...
W20260620 21:54:46.663962 775 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:46 dist-test-slave-kk5c krb5kdc[21693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:54:46.792924 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:46.905115 21706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:46.905411 21706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:46.905454 21706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:46.909556 21706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:46.909628 21706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:46.909649 21706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:46.909665 21706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:46.909682 21706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:46.915078 21706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:46.916204 21706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:46.917163 21706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:46.926614 21714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:46.926608 21711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:46.926597 21712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:46.926841 21706 server_base.cc:1034] running on GCE node
I20260620 21:54:46.927623 21706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:46.929003 21706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:46.930186 21706 hybrid_clock.cc:648] HybridClock initialized: now 1781992486930150 us; error 55 us; skew 500 ppm
Jun 20 21:54:46 dist-test-slave-kk5c krb5kdc[21693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:46.936906 21706 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:54:46.937947 21706 webserver.cc:460] Webserver started at http://127.0.0.1:39729/ using document root <none> and password file <none>
I20260620 21:54:46.938555 21706 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:46.938655 21706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:46.938956 21706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:46.941208 21706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "13095025e26b4902959166ab093c35f6"
format_stamp: "Formatted at 2026-06-20 21:54:46 on dist-test-slave-kk5c"
server_key: "0849c08bb67a903760ddc529fb592e82"
server_key_iv: "795b6c2681e6d61f734fab3b5eae561e"
server_key_version: "encryptionkey@0"
I20260620 21:54:46.941814 21706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "13095025e26b4902959166ab093c35f6"
format_stamp: "Formatted at 2026-06-20 21:54:46 on dist-test-slave-kk5c"
server_key: "0849c08bb67a903760ddc529fb592e82"
server_key_iv: "795b6c2681e6d61f734fab3b5eae561e"
server_key_version: "encryptionkey@0"
I20260620 21:54:46.945698 21706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 21:54:46.948261 21721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:46.949592 21706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 21:54:46.949756 21706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "13095025e26b4902959166ab093c35f6"
format_stamp: "Formatted at 2026-06-20 21:54:46 on dist-test-slave-kk5c"
server_key: "0849c08bb67a903760ddc529fb592e82"
server_key_iv: "795b6c2681e6d61f734fab3b5eae561e"
server_key_version: "encryptionkey@0"
I20260620 21:54:46.949885 21706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:46.966998 21706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:46.970238 21706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:46.970455 21706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:46.977874 21706 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39315
I20260620 21:54:46.977900 21773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39315 every 8 connection(s)
I20260620 21:54:46.978830 21706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:46.979732 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 21706
I20260620 21:54:46.979908 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:46.980212 775 external_mini_cluster.cc:1428] Setting key 2263eaa19c50ba1d4af7ef03d17304a8
I20260620 21:54:46.981873 21774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:46.988514 21774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6: Bootstrap starting.
Jun 20 21:54:46 dist-test-slave-kk5c krb5kdc[21693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 21:54:46.991173 21774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:46.992125 21774 log.cc:824] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:46.994194 21774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6: No bootstrap required, opened a new log
I20260620 21:54:46.995574 21777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:46.981594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:45255 (local address 127.0.0.1:39315)
0620 21:54:46.982002 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:46.982020 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:46.982062 (+ 42us) server_negotiation.cc:407] Connection header received
0620 21:54:46.982872 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:46.982895 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:46.983355 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:46.983712 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:46.984371 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:46.985887 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:46.986770 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:46.987495 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:46.987677 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:46.990494 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:46.990500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:46.990516 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:46.990548 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:46.992588 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:46.993138 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:46.993151 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:46.993155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:46.993263 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:46.993631 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:46.993636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:46.993638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:46.994022 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:46.994234 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:46.994566 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 21:54:46.994787 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260620 21:54:46.998898 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:46.999444 21774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13095025e26b4902959166ab093c35f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39315 } }
I20260620 21:54:46.999734 21774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:46.999835 21774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13095025e26b4902959166ab093c35f6, State: Initialized, Role: FOLLOWER
I20260620 21:54:47.000329 21774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13095025e26b4902959166ab093c35f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39315 } }
I20260620 21:54:47.000497 21774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:47.000569 21774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:47.000672 21774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:47.002171 21774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13095025e26b4902959166ab093c35f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39315 } }
I20260620 21:54:47.002588 21774 leader_election.cc:302] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13095025e26b4902959166ab093c35f6; no voters:
I20260620 21:54:47.002967 21774 leader_election.cc:288] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:47.003070 21784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:47.003324 21784 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [term 1 LEADER]: Becoming Leader. State: Replica: 13095025e26b4902959166ab093c35f6, State: Running, Role: LEADER
I20260620 21:54:47.003726 21784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13095025e26b4902959166ab093c35f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39315 } }
I20260620 21:54:47.004362 21774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:47.004959 21784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13095025e26b4902959166ab093c35f6. Latest consensus state: current_term: 1 leader_uuid: "13095025e26b4902959166ab093c35f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13095025e26b4902959166ab093c35f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39315 } } }
I20260620 21:54:47.005075 21784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:47.005453 21789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:47.007236 21785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13095025e26b4902959166ab093c35f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13095025e26b4902959166ab093c35f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39315 } } }
I20260620 21:54:47.007382 21785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:47.008630 21789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:47.011632 21777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:47.001104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:52153 (local address 127.0.0.1:39315)
0620 21:54:47.001225 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:47.001230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:47.001387 (+ 157us) server_negotiation.cc:407] Connection header received
0620 21:54:47.001510 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:47.001513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:47.001553 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:47.001618 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:47.002574 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:47.003290 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:47.004446 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:47.004842 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:47.004956 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:47.007109 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:47.007113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:47.007116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:47.007140 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:47.009193 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:47.009769 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:47.009773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:47.009775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:47.009873 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:47.010252 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:47.010255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:47.010257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:47.010489 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:47.010647 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:47.011355 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 21:54:47.011464 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:54:47.016917 21789 catalog_manager.cc:1347] Generated new cluster ID: 67195a9903254984a77d5fedc422de09
I20260620 21:54:47.016990 21789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:47.026127 21789 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:47.027135 21789 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:47.040335 21789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13095025e26b4902959166ab093c35f6: Generated new TSK 0
I20260620 21:54:47.041132 21789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:47.044615 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 21706
W20260620 21:54:47.050275 21691 connection.cc:535] client connection to 127.0.0.1:39315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39315): Connection reset by peer (error 104)
2026-06-20T21:54:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:47 dist-test-slave-kk5c krb5kdc[21805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:47 dist-test-slave-kk5c krb5kdc[21805](info): set up 2 sockets
Jun 20 21:54:47 dist-test-slave-kk5c krb5kdc[21805](info): commencing operation
krb5kdc: starting...
W20260620 21:54:48.111673 775 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:48 dist-test-slave-kk5c krb5kdc[21805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:54:48.262699 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39707
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:48.380198 21818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:48.380545 21818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:48.380612 21818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:48.384917 21818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:48.385032 21818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:48.385082 21818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:48.385141 21818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:48.385178 21818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:48.391764 21818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:48.392966 21818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:48.394050 21818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:48.403574 21823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:48.403725 21818 server_base.cc:1034] running on GCE node
W20260620 21:54:48.403565 21826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:48.403568 21824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:48.404448 21818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:48.405654 21818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:48.406868 21818 hybrid_clock.cc:648] HybridClock initialized: now 1781992488406832 us; error 50 us; skew 500 ppm
Jun 20 21:54:48 dist-test-slave-kk5c krb5kdc[21805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:48.409970 21818 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:54:48.410955 21818 webserver.cc:460] Webserver started at http://127.0.0.1:37867/ using document root <none> and password file <none>
I20260620 21:54:48.411639 21818 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:48.411739 21818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:48.411985 21818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:48.414184 21818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "8664af9dade34a5481ecebacf9b6d322"
format_stamp: "Formatted at 2026-06-20 21:54:48 on dist-test-slave-kk5c"
server_key: "8935e3a9900e53ed3a8edbb62aa0236c"
server_key_iv: "432825d1cf195f9d2d8e696880c0a033"
server_key_version: "encryptionkey@0"
I20260620 21:54:48.414815 21818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "8664af9dade34a5481ecebacf9b6d322"
format_stamp: "Formatted at 2026-06-20 21:54:48 on dist-test-slave-kk5c"
server_key: "8935e3a9900e53ed3a8edbb62aa0236c"
server_key_iv: "432825d1cf195f9d2d8e696880c0a033"
server_key_version: "encryptionkey@0"
I20260620 21:54:48.418881 21818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 21:54:48.421792 21833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:48.423429 21818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 21:54:48.423595 21818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "8664af9dade34a5481ecebacf9b6d322"
format_stamp: "Formatted at 2026-06-20 21:54:48 on dist-test-slave-kk5c"
server_key: "8935e3a9900e53ed3a8edbb62aa0236c"
server_key_iv: "432825d1cf195f9d2d8e696880c0a033"
server_key_version: "encryptionkey@0"
I20260620 21:54:48.423740 21818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:48.441342 21818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:48.445688 21818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:48.445963 21818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:48.455271 21818 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39707
I20260620 21:54:48.455576 21885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39707 every 8 connection(s)
I20260620 21:54:48.456431 21818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:48.460309 21886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:48.460628 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 21818
I20260620 21:54:48.460917 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:48.461383 775 external_mini_cluster.cc:1428] Setting key a31fc983ba2479c710a4f19c008a0946
I20260620 21:54:48.468379 21886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322: Bootstrap starting.
I20260620 21:54:48.470880 21886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:48.471837 21886 log.cc:824] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:48.472908 21889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:48.462828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:38517 (local address 127.0.0.1:39707)
0620 21:54:48.463342 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:48.463358 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:48.463395 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:54:48.464398 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:48.464422 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:48.464884 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:48.465256 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:48.465723 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:48.466815 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:48.468038 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:48.468640 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:48.468840 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:48.469875 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:48.469881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:48.469895 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:48.470456 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:48.470516 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:48.470543 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:48.470823 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:48.470969 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 21:54:48.471489 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":137,"threads_started":1}
I20260620 21:54:48.474244 21886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322: No bootstrap required, opened a new log
I20260620 21:54:48.476670 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:48.478708 21886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8664af9dade34a5481ecebacf9b6d322" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39707 } }
I20260620 21:54:48.479084 21886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:48.479177 21886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8664af9dade34a5481ecebacf9b6d322, State: Initialized, Role: FOLLOWER
I20260620 21:54:48.479719 21886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8664af9dade34a5481ecebacf9b6d322" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39707 } }
I20260620 21:54:48.479871 21886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:48.479949 21886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:48.480086 21886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:48.481114 21886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8664af9dade34a5481ecebacf9b6d322" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39707 } }
I20260620 21:54:48.481491 21886 leader_election.cc:302] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8664af9dade34a5481ecebacf9b6d322; no voters:
I20260620 21:54:48.481853 21886 leader_election.cc:288] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:48.481936 21896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:48.482146 21896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [term 1 LEADER]: Becoming Leader. State: Replica: 8664af9dade34a5481ecebacf9b6d322, State: Running, Role: LEADER
I20260620 21:54:48.482472 21896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8664af9dade34a5481ecebacf9b6d322" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39707 } }
I20260620 21:54:48.483336 21886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:48.483829 21889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:48.478843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:58313 (local address 127.0.0.1:39707)
0620 21:54:48.478993 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:48.478998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:48.479183 (+ 185us) server_negotiation.cc:407] Connection header received
0620 21:54:48.479375 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:48.479378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:48.479433 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:48.479509 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:48.480285 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:48.480808 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:48.481868 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:48.482228 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:48.482761 (+ 533us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:48.483351 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:48.483354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:48.483357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:48.483371 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:48.483396 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:48.483431 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:48.483521 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:48.483644 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 21:54:48.483681 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:54:48.483917 21896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8664af9dade34a5481ecebacf9b6d322" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8664af9dade34a5481ecebacf9b6d322" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39707 } } }
I20260620 21:54:48.484032 21896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:48.484988 21897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8664af9dade34a5481ecebacf9b6d322. Latest consensus state: current_term: 1 leader_uuid: "8664af9dade34a5481ecebacf9b6d322" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8664af9dade34a5481ecebacf9b6d322" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39707 } } }
I20260620 21:54:48.485141 21897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:48.485457 21901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:48.488663 21901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:48.495733 21901 catalog_manager.cc:1347] Generated new cluster ID: 3c31f2da2f1b47de96aa6573a82e835c
I20260620 21:54:48.495822 21901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:48.509382 21901 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:48.510497 21901 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:48.516929 21901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8664af9dade34a5481ecebacf9b6d322: Generated new TSK 0
I20260620 21:54:48.517696 21901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:48.529552 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 21818
W20260620 21:54:48.535410 21803 connection.cc:535] client connection to 127.0.0.1:39707 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39707): Connection reset by peer (error 104)
2026-06-20T21:54:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1485 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:48 dist-test-slave-kk5c krb5kdc[21917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:48 dist-test-slave-kk5c krb5kdc[21917](info): set up 2 sockets
Jun 20 21:54:48 dist-test-slave-kk5c krb5kdc[21917](info): commencing operation
krb5kdc: starting...
W20260620 21:54:49.587831 775 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:49 dist-test-slave-kk5c krb5kdc[21917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:54:49.727676 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:49.847440 21930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:49.847729 21930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:49.847774 21930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:49.851861 21930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:49.851922 21930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:49.851943 21930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:49.851958 21930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:49.851974 21930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:49.857601 21930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:49.858712 21930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:49.859750 21930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:49.868860 21935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:49.868825 21936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:49.868831 21938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:49.869273 21930 server_base.cc:1034] running on GCE node
I20260620 21:54:49.869976 21930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:49.871465 21930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:49.872675 21930 hybrid_clock.cc:648] HybridClock initialized: now 1781992489872662 us; error 32 us; skew 500 ppm
Jun 20 21:54:49 dist-test-slave-kk5c krb5kdc[21917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:49.876204 21930 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:54:49.877269 21930 webserver.cc:460] Webserver started at http://127.0.0.1:44629/ using document root <none> and password file <none>
I20260620 21:54:49.877866 21930 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:49.877964 21930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:49.878223 21930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:49.880792 21930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "e791056f619e431286813bde42e0e655"
format_stamp: "Formatted at 2026-06-20 21:54:49 on dist-test-slave-kk5c"
server_key: "2de347473af9959414d3e9cf419e69ee"
server_key_iv: "f430d29f6cafae24f5425fcab7349958"
server_key_version: "encryptionkey@0"
I20260620 21:54:49.881417 21930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "e791056f619e431286813bde42e0e655"
format_stamp: "Formatted at 2026-06-20 21:54:49 on dist-test-slave-kk5c"
server_key: "2de347473af9959414d3e9cf419e69ee"
server_key_iv: "f430d29f6cafae24f5425fcab7349958"
server_key_version: "encryptionkey@0"
I20260620 21:54:49.885491 21930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:54:49.888386 21945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:49.889950 21930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:49.890324 21930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "e791056f619e431286813bde42e0e655"
format_stamp: "Formatted at 2026-06-20 21:54:49 on dist-test-slave-kk5c"
server_key: "2de347473af9959414d3e9cf419e69ee"
server_key_iv: "f430d29f6cafae24f5425fcab7349958"
server_key_version: "encryptionkey@0"
I20260620 21:54:49.890516 21930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:49.895889 21930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:49.899497 21930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:49.899717 21930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:49.907087 21930 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43825
I20260620 21:54:49.907104 21997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43825 every 8 connection(s)
I20260620 21:54:49.908128 21930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:49.911080 21998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:49.914952 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 21930
I20260620 21:54:49.915130 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:49.915452 775 external_mini_cluster.cc:1428] Setting key 07c96d6d10d3bfbe3ef9c3e56bb443c4
I20260620 21:54:49.917153 21998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655: Bootstrap starting.
I20260620 21:54:49.919848 21998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:49.920792 21998 log.cc:824] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:49.921064 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:49.916928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:34927 (local address 127.0.0.1:43825)
0620 21:54:49.917387 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:49.917402 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:49.917438 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:54:49.918071 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:49.918088 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:49.918394 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:49.918686 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:49.919649 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:49.919654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:49.919666 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:49.919677 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:49.919756 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:49.919780 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:49.919876 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:49.920095 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 21:54:49.920243 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":146,"threads_started":1}
I20260620 21:54:49.922873 21998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655: No bootstrap required, opened a new log
I20260620 21:54:49.924144 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:49.927824 21998 raft_consensus.cc:348] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e791056f619e431286813bde42e0e655" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43825 } }
I20260620 21:54:49.928171 21998 raft_consensus.cc:374] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:49.928246 21998 raft_consensus.cc:729] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e791056f619e431286813bde42e0e655, State: Initialized, Role: FOLLOWER
I20260620 21:54:49.928563 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:49.926766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:54371 (local address 127.0.0.1:43825)
0620 21:54:49.926895 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:49.926900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:49.927080 (+ 180us) server_negotiation.cc:407] Connection header received
0620 21:54:49.927219 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:49.927222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:49.927262 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:49.927343 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:49.928123 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:49.928130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:49.928135 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:49.928149 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:49.928178 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:49.928194 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:49.928289 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:49.928404 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 21:54:49.928445 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:54:49.928769 21998 consensus_queue.cc:260] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e791056f619e431286813bde42e0e655" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43825 } }
I20260620 21:54:49.928934 21998 raft_consensus.cc:388] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:49.929003 21998 raft_consensus.cc:482] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:49.929105 21998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:49.930181 21998 raft_consensus.cc:504] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e791056f619e431286813bde42e0e655" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43825 } }
I20260620 21:54:49.930554 21998 leader_election.cc:302] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e791056f619e431286813bde42e0e655; no voters:
I20260620 21:54:49.930900 21998 leader_election.cc:288] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:49.931039 22008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:49.931353 22008 raft_consensus.cc:686] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [term 1 LEADER]: Becoming Leader. State: Replica: e791056f619e431286813bde42e0e655, State: Running, Role: LEADER
I20260620 21:54:49.931705 22008 consensus_queue.cc:237] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e791056f619e431286813bde42e0e655" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43825 } }
I20260620 21:54:49.932296 21998 sys_catalog.cc:564] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:49.933862 22010 sys_catalog.cc:455] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e791056f619e431286813bde42e0e655. Latest consensus state: current_term: 1 leader_uuid: "e791056f619e431286813bde42e0e655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e791056f619e431286813bde42e0e655" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43825 } } }
I20260620 21:54:49.933827 22009 sys_catalog.cc:455] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e791056f619e431286813bde42e0e655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e791056f619e431286813bde42e0e655" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43825 } } }
I20260620 21:54:49.933990 22009 sys_catalog.cc:458] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:49.934290 22010 sys_catalog.cc:458] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:49.934540 22020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:49.937731 22020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:49.943789 22020 catalog_manager.cc:1347] Generated new cluster ID: fa14749c96b24bf2b8fcfb9e8d358fb0
I20260620 21:54:49.943886 22020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:49.969873 22020 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:49.970254 22020 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:49.974977 22020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e791056f619e431286813bde42e0e655: Generated new TSK 0
I20260620 21:54:49.975767 22020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:49.982044 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 21930
2026-06-20T21:54:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:50 dist-test-slave-kk5c krb5kdc[22029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:50 dist-test-slave-kk5c krb5kdc[22029](info): set up 2 sockets
Jun 20 21:54:50 dist-test-slave-kk5c krb5kdc[22029](info): commencing operation
krb5kdc: starting...
W20260620 21:54:51.063519 775 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:51 dist-test-slave-kk5c krb5kdc[22029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:54:51.195048 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:51.309206 22042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:51.309511 22042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:51.309556 22042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:51.313817 22042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:51.313895 22042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:51.313915 22042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:51.313932 22042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:51.313948 22042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:51.319801 22042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34851
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:51.321023 22042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:51.322086 22042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:51.331147 22047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:51.331386 22042 server_base.cc:1034] running on GCE node
W20260620 21:54:51.331149 22048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:51.331149 22050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:51.332335 22042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:51.333519 22042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:51.334729 22042 hybrid_clock.cc:648] HybridClock initialized: now 1781992491334717 us; error 27 us; skew 500 ppm
Jun 20 21:54:51 dist-test-slave-kk5c krb5kdc[22029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:51.337764 22042 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:54:51.338878 22042 webserver.cc:460] Webserver started at http://127.0.0.1:41053/ using document root <none> and password file <none>
I20260620 21:54:51.339537 22042 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:51.339634 22042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:51.339860 22042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:51.342170 22042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "7855db2a637f4471b747eac5ca625142"
format_stamp: "Formatted at 2026-06-20 21:54:51 on dist-test-slave-kk5c"
server_key: "ebe5a6d1171ad654a61067132632a42e"
server_key_iv: "1b19b488dd73d1bff888ef8351b647f7"
server_key_version: "encryptionkey@0"
I20260620 21:54:51.342765 22042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "7855db2a637f4471b747eac5ca625142"
format_stamp: "Formatted at 2026-06-20 21:54:51 on dist-test-slave-kk5c"
server_key: "ebe5a6d1171ad654a61067132632a42e"
server_key_iv: "1b19b488dd73d1bff888ef8351b647f7"
server_key_version: "encryptionkey@0"
I20260620 21:54:51.346815 22042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 21:54:51.349661 22057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:51.351033 22042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:54:51.351249 22042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "7855db2a637f4471b747eac5ca625142"
format_stamp: "Formatted at 2026-06-20 21:54:51 on dist-test-slave-kk5c"
server_key: "ebe5a6d1171ad654a61067132632a42e"
server_key_iv: "1b19b488dd73d1bff888ef8351b647f7"
server_key_version: "encryptionkey@0"
I20260620 21:54:51.351384 22042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:51.372115 22042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:51.375396 22042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:51.375567 22042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:51.382956 22042 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34851
I20260620 21:54:51.382968 22109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34851 every 8 connection(s)
I20260620 21:54:51.384017 22042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:51.387051 22110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:51.392472 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 22042
I20260620 21:54:51.392609 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:51.392910 775 external_mini_cluster.cc:1428] Setting key c1cf8cfb3d30fc7e8c3a4d390c188e04
I20260620 21:54:51.393421 22110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142: Bootstrap starting.
I20260620 21:54:51.396147 22110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:51.397362 22110 log.cc:824] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:51.400002 22110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142: No bootstrap required, opened a new log
Jun 20 21:54:51 dist-test-slave-kk5c krb5kdc[22029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 21:54:51.405054 22110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7855db2a637f4471b747eac5ca625142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } }
I20260620 21:54:51.405359 22110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:51.405439 22110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7855db2a637f4471b747eac5ca625142, State: Initialized, Role: FOLLOWER
I20260620 21:54:51.405975 22110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7855db2a637f4471b747eac5ca625142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } }
I20260620 21:54:51.406176 22110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:51.406252 22110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:51.406411 22110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:51.407564 22110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7855db2a637f4471b747eac5ca625142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } }
I20260620 21:54:51.407989 22110 leader_election.cc:302] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7855db2a637f4471b747eac5ca625142; no voters:
I20260620 21:54:51.408398 22110 leader_election.cc:288] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:51.408502 22115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:51.408742 22115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [term 1 LEADER]: Becoming Leader. State: Replica: 7855db2a637f4471b747eac5ca625142, State: Running, Role: LEADER
I20260620 21:54:51.409132 22115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7855db2a637f4471b747eac5ca625142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } }
I20260620 21:54:51.409940 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:51.394412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:59609 (local address 127.0.0.1:34851)
0620 21:54:51.394860 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:51.394876 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:51.394912 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:54:51.395847 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:51.395869 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:51.396305 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:51.396621 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:51.397628 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:51.398947 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:51.400320 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:51.400804 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:51.400940 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:51.404191 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:51.404197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:51.404214 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:51.404255 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:51.406541 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:51.407141 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:51.407147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:51.407154 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:51.407350 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:51.407718 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:51.407721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:51.407723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:51.408207 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:51.408460 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:51.408784 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 21:54:51.409047 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":138,"threads_started":1}
I20260620 21:54:51.410413 22117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7855db2a637f4471b747eac5ca625142. Latest consensus state: current_term: 1 leader_uuid: "7855db2a637f4471b747eac5ca625142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7855db2a637f4471b747eac5ca625142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } } }
I20260620 21:54:51.409940 22110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:51.410609 22116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7855db2a637f4471b747eac5ca625142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7855db2a637f4471b747eac5ca625142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } } }
I20260620 21:54:51.410935 22116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:51.410728 22117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:51.411525 22119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:51.414430 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:51.416453 22119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:51.422546 22119 catalog_manager.cc:1347] Generated new cluster ID: a6801e6dc2fb48c2a373c29bd6e2be8c
I20260620 21:54:51.422665 22119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:51.429859 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:51.418603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:35067 (local address 127.0.0.1:34851)
0620 21:54:51.418717 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:51.418722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:51.419428 (+ 706us) server_negotiation.cc:407] Connection header received
0620 21:54:51.419668 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:51.419672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:51.419731 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:51.419789 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:51.420879 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:51.421783 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:51.423087 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:51.423516 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:51.423600 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:51.425714 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:51.425718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:51.425724 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:51.425757 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:51.427718 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:51.428347 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:51.428351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:51.428354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:51.428457 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:51.428829 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:51.428833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:51.428835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:51.429082 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:51.429290 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:51.429565 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 21:54:51.429681 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 21:54:51.448727 22119 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:51.449795 22119 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:51.454840 22119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7855db2a637f4471b747eac5ca625142: Generated new TSK 0
I20260620 21:54:51.455737 22119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:51.467338 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 22042
W20260620 21:54:51.473140 22027 connection.cc:535] client connection to 127.0.0.1:34851 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34851): Connection reset by peer (error 104)
2026-06-20T21:54:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:51 dist-test-slave-kk5c krb5kdc[22141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:51 dist-test-slave-kk5c krb5kdc[22141](info): set up 2 sockets
Jun 20 21:54:51 dist-test-slave-kk5c krb5kdc[22141](info): commencing operation
krb5kdc: starting...
W20260620 21:54:52.547535 775 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:52 dist-test-slave-kk5c krb5kdc[22141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:54:52.696218 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:52.810576 22154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:52.810917 22154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:52.810995 22154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:52.815164 22154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:52.815301 22154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:52.815340 22154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:52.815373 22154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:52.815402 22154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:52.821137 22154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35999
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:52.822309 22154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:52.823405 22154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:52.833004 22162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:52.832999 22160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:52.833170 22154 server_base.cc:1034] running on GCE node
W20260620 21:54:52.833405 22159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:52.834052 22154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:52.835246 22154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:52.836447 22154 hybrid_clock.cc:648] HybridClock initialized: now 1781992492836435 us; error 32 us; skew 500 ppm
Jun 20 21:54:52 dist-test-slave-kk5c krb5kdc[22141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:52.844923 22154 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:54:52.845889 22154 webserver.cc:460] Webserver started at http://127.0.0.1:38373/ using document root <none> and password file <none>
I20260620 21:54:52.846448 22154 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:52.846511 22154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:52.846683 22154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:52.848922 22154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "0134a3de328f4645b1aefbf6b6bd5cca"
format_stamp: "Formatted at 2026-06-20 21:54:52 on dist-test-slave-kk5c"
server_key: "51d879b585bd341e7c0419b100d7db80"
server_key_iv: "b8b2b884e9d54f36465c963ec43e26ed"
server_key_version: "encryptionkey@0"
I20260620 21:54:52.849484 22154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "0134a3de328f4645b1aefbf6b6bd5cca"
format_stamp: "Formatted at 2026-06-20 21:54:52 on dist-test-slave-kk5c"
server_key: "51d879b585bd341e7c0419b100d7db80"
server_key_iv: "b8b2b884e9d54f36465c963ec43e26ed"
server_key_version: "encryptionkey@0"
I20260620 21:54:52.853603 22154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 21:54:52.856256 22169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:52.857681 22154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 21:54:52.857848 22154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "0134a3de328f4645b1aefbf6b6bd5cca"
format_stamp: "Formatted at 2026-06-20 21:54:52 on dist-test-slave-kk5c"
server_key: "51d879b585bd341e7c0419b100d7db80"
server_key_iv: "b8b2b884e9d54f36465c963ec43e26ed"
server_key_version: "encryptionkey@0"
I20260620 21:54:52.857987 22154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:52.873739 22154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:52.877362 22154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:52.877610 22154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:52.885283 22154 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35999
I20260620 21:54:52.885375 22221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35999 every 8 connection(s)
I20260620 21:54:52.886356 22154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:52.889524 22222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:52.893754 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 22154
I20260620 21:54:52.893932 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:52.894273 775 external_mini_cluster.cc:1428] Setting key 7bf2539faf971e34562e339b2afdf1aa
I20260620 21:54:52.896397 22222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca: Bootstrap starting.
I20260620 21:54:52.898921 22222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:52.899832 22222 log.cc:824] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:52.902658 22222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca: No bootstrap required, opened a new log
I20260620 21:54:52.905049 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:52.895823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:39269 (local address 127.0.0.1:35999)
0620 21:54:52.896346 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:52.896362 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:52.896398 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:54:52.897356 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:52.897377 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:52.897770 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:52.898079 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:52.898461 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:52.899668 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:52.900907 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:52.901401 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:52.901521 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:52.902755 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:52.902766 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:52.902787 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:52.903316 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:52.903376 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:52.903402 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:52.903518 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:52.903803 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 21:54:52.903963 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":134,"threads_started":1}
I20260620 21:54:52.906303 22222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0134a3de328f4645b1aefbf6b6bd5cca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35999 } }
I20260620 21:54:52.906567 22222 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:52.906607 22222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0134a3de328f4645b1aefbf6b6bd5cca, State: Initialized, Role: FOLLOWER
I20260620 21:54:52.907225 22222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0134a3de328f4645b1aefbf6b6bd5cca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35999 } }
I20260620 21:54:52.907404 22222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:52.907481 22222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:52.907595 22222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:52.908355 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:52.908857 22222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0134a3de328f4645b1aefbf6b6bd5cca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35999 } }
I20260620 21:54:52.909243 22222 leader_election.cc:302] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0134a3de328f4645b1aefbf6b6bd5cca; no voters:
I20260620 21:54:52.909626 22222 leader_election.cc:288] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:52.909719 22227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:52.909940 22227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [term 1 LEADER]: Becoming Leader. State: Replica: 0134a3de328f4645b1aefbf6b6bd5cca, State: Running, Role: LEADER
I20260620 21:54:52.910355 22227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0134a3de328f4645b1aefbf6b6bd5cca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35999 } }
I20260620 21:54:52.911152 22222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:52.912393 22232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0134a3de328f4645b1aefbf6b6bd5cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0134a3de328f4645b1aefbf6b6bd5cca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35999 } } }
I20260620 21:54:52.912559 22232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:52.913000 22241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:52.912954 22231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0134a3de328f4645b1aefbf6b6bd5cca. Latest consensus state: current_term: 1 leader_uuid: "0134a3de328f4645b1aefbf6b6bd5cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0134a3de328f4645b1aefbf6b6bd5cca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35999 } } }
I20260620 21:54:52.913046 22231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:52.916764 22241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:52.917229 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:52.910821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:56033 (local address 127.0.0.1:35999)
0620 21:54:52.910982 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:52.910987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:52.911505 (+ 518us) server_negotiation.cc:407] Connection header received
0620 21:54:52.911712 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:52.911716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:52.911769 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:52.911881 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:52.912351 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:52.913280 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:52.914532 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:52.914897 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:52.915009 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:52.916634 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:52.916638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:52.916641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:52.916656 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:52.916690 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:52.916705 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:52.916821 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:52.917044 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 21:54:52.917087 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 21:54:52.923321 22241 catalog_manager.cc:1347] Generated new cluster ID: e2616c8fafe94a6c8f276822d4bd7b69
I20260620 21:54:52.923465 22241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:52.938994 22241 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:52.940027 22241 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:52.945159 22241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0134a3de328f4645b1aefbf6b6bd5cca: Generated new TSK 0
I20260620 21:54:52.945879 22241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:52.956465 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 22154
W20260620 21:54:52.961755 22139 connection.cc:535] client connection to 127.0.0.1:35999 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35999): Connection reset by peer (error 104)
2026-06-20T21:54:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1488 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:53 dist-test-slave-kk5c krb5kdc[22253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:53 dist-test-slave-kk5c krb5kdc[22253](info): set up 2 sockets
Jun 20 21:54:53 dist-test-slave-kk5c krb5kdc[22253](info): commencing operation
krb5kdc: starting...
W20260620 21:54:54.043592 775 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:54 dist-test-slave-kk5c krb5kdc[22253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 21:54:54.179505 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41687
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:54.294570 22266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:54.294883 22266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:54.294927 22266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:54.299121 22266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:54.299274 22266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:54.299312 22266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:54.299331 22266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:54.299348 22266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:54.305064 22266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:54.306265 22266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:54.307372 22266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:54.316576 22272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:54.316548 22271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:54.316548 22274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:54.317360 22266 server_base.cc:1034] running on GCE node
I20260620 21:54:54.318120 22266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:54.319630 22266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:54.320860 22266 hybrid_clock.cc:648] HybridClock initialized: now 1781992494320844 us; error 35 us; skew 500 ppm
Jun 20 21:54:54 dist-test-slave-kk5c krb5kdc[22253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:54.324301 22266 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 21:54:54.325273 22266 webserver.cc:460] Webserver started at http://127.0.0.1:43493/ using document root <none> and password file <none>
I20260620 21:54:54.325846 22266 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:54.325910 22266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:54.326093 22266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:54.328296 22266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "55715fbd2d0d42b888ce1815e1eac7df"
format_stamp: "Formatted at 2026-06-20 21:54:54 on dist-test-slave-kk5c"
server_key: "8ae9eabd91ede47d9b91ed1dd7092815"
server_key_iv: "7509f313397ba9b7e8fc49d5621c380e"
server_key_version: "encryptionkey@0"
I20260620 21:54:54.328840 22266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "55715fbd2d0d42b888ce1815e1eac7df"
format_stamp: "Formatted at 2026-06-20 21:54:54 on dist-test-slave-kk5c"
server_key: "8ae9eabd91ede47d9b91ed1dd7092815"
server_key_iv: "7509f313397ba9b7e8fc49d5621c380e"
server_key_version: "encryptionkey@0"
I20260620 21:54:54.332727 22266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:54:54.335453 22281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:54.336891 22266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 21:54:54.337013 22266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "55715fbd2d0d42b888ce1815e1eac7df"
format_stamp: "Formatted at 2026-06-20 21:54:54 on dist-test-slave-kk5c"
server_key: "8ae9eabd91ede47d9b91ed1dd7092815"
server_key_iv: "7509f313397ba9b7e8fc49d5621c380e"
server_key_version: "encryptionkey@0"
I20260620 21:54:54.337131 22266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:54.341883 22266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:54.345242 22266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:54.345397 22266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:54.352147 22266 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41687
I20260620 21:54:54.352224 22333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41687 every 8 connection(s)
I20260620 21:54:54.353124 22266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:54.356046 22334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:54.356478 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 22266
I20260620 21:54:54.356671 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:54.357013 775 external_mini_cluster.cc:1428] Setting key a0c3c097bbc7ce57b1bbc737fd23023f
I20260620 21:54:54.362579 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:54.358434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:49851 (local address 127.0.0.1:41687)
0620 21:54:54.358913 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:54.358927 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:54.358994 (+ 67us) server_negotiation.cc:407] Connection header received
0620 21:54:54.359782 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:54.359804 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:54.360116 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:54.360481 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:54.361348 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:54.361353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:54.361365 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:54.361376 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:54.361432 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:54.361454 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:54.361621 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:54.361693 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 21:54:54.361861 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":159,"threads_started":1}
I20260620 21:54:54.363545 22334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df: Bootstrap starting.
I20260620 21:54:54.365553 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:54.366006 22334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:54.366770 22334 log.cc:824] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:54.369381 22334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df: No bootstrap required, opened a new log
I20260620 21:54:54.369529 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:54.367776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:58135 (local address 127.0.0.1:41687)
0620 21:54:54.367914 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:54.367919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:54.368049 (+ 130us) server_negotiation.cc:407] Connection header received
0620 21:54:54.368156 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:54.368159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:54.368197 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:54.368287 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:54.369064 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:54.369068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:54.369071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:54.369083 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:54.369113 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:54.369126 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:54.369222 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:54.369360 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 21:54:54.369403 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:54:54.373261 22334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55715fbd2d0d42b888ce1815e1eac7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41687 } }
I20260620 21:54:54.373574 22334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:54.373620 22334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55715fbd2d0d42b888ce1815e1eac7df, State: Initialized, Role: FOLLOWER
I20260620 21:54:54.374053 22334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55715fbd2d0d42b888ce1815e1eac7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41687 } }
I20260620 21:54:54.374181 22334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:54.374230 22334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:54.374305 22334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:54.375329 22334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55715fbd2d0d42b888ce1815e1eac7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41687 } }
I20260620 21:54:54.375658 22334 leader_election.cc:302] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55715fbd2d0d42b888ce1815e1eac7df; no voters:
I20260620 21:54:54.376034 22334 leader_election.cc:288] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:54.376199 22344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:54.376443 22344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [term 1 LEADER]: Becoming Leader. State: Replica: 55715fbd2d0d42b888ce1815e1eac7df, State: Running, Role: LEADER
I20260620 21:54:54.376904 22344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55715fbd2d0d42b888ce1815e1eac7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41687 } }
I20260620 21:54:54.377408 22334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:54.378744 22345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55715fbd2d0d42b888ce1815e1eac7df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55715fbd2d0d42b888ce1815e1eac7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41687 } } }
I20260620 21:54:54.378788 22346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55715fbd2d0d42b888ce1815e1eac7df. Latest consensus state: current_term: 1 leader_uuid: "55715fbd2d0d42b888ce1815e1eac7df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55715fbd2d0d42b888ce1815e1eac7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41687 } } }
I20260620 21:54:54.378872 22345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:54.378890 22346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:54.379298 22353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:54.382645 22353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:54.388764 22353 catalog_manager.cc:1347] Generated new cluster ID: 911c20ed01bc4a049850208abd39e5b5
I20260620 21:54:54.388899 22353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:54.398787 22353 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:54.399140 22353 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:54.413643 22353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55715fbd2d0d42b888ce1815e1eac7df: Generated new TSK 0
I20260620 21:54:54.414582 22353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:54.427121 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 22266
2026-06-20T21:54:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:54 dist-test-slave-kk5c krb5kdc[22365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:54 dist-test-slave-kk5c krb5kdc[22365](info): set up 2 sockets
Jun 20 21:54:54 dist-test-slave-kk5c krb5kdc[22365](info): commencing operation
krb5kdc: starting...
W20260620 21:54:55.503641 775 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:55 dist-test-slave-kk5c krb5kdc[22365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:54:55.640012 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:42339
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:42339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:55.754004 22378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:55.754287 22378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:55.754331 22378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:55.758493 22378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:55.758574 22378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:55.758594 22378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:55.758610 22378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:55.758626 22378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:55.764161 22378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:42339
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:42339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:55.765324 22378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:55.766462 22378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:55.775517 22383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:55.775643 22384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:55.775573 22386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:55.775847 22378 server_base.cc:1034] running on GCE node
I20260620 21:54:55.776770 22378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:55.778100 22378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:55.779291 22378 hybrid_clock.cc:648] HybridClock initialized: now 1781992495779269 us; error 38 us; skew 500 ppm
Jun 20 21:54:55 dist-test-slave-kk5c krb5kdc[22365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:55.782464 22378 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:54:55.783700 22378 webserver.cc:460] Webserver started at http://127.0.193.254:40223/ using document root <none> and password file <none>
I20260620 21:54:55.784348 22378 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:55.784457 22378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:55.784734 22378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:55.787158 22378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "aecbef5ba11c4d539609293b0b186ec9"
format_stamp: "Formatted at 2026-06-20 21:54:55 on dist-test-slave-kk5c"
server_key: "f14c6e98b5caf4712bb64e741a698125"
server_key_iv: "5526bf2d851ab51275cfe99ab05accde"
server_key_version: "encryptionkey@0"
I20260620 21:54:55.787963 22378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "aecbef5ba11c4d539609293b0b186ec9"
format_stamp: "Formatted at 2026-06-20 21:54:55 on dist-test-slave-kk5c"
server_key: "f14c6e98b5caf4712bb64e741a698125"
server_key_iv: "5526bf2d851ab51275cfe99ab05accde"
server_key_version: "encryptionkey@0"
I20260620 21:54:55.793048 22378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 21:54:55.796466 22393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:55.798110 22378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 21:54:55.798334 22378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "aecbef5ba11c4d539609293b0b186ec9"
format_stamp: "Formatted at 2026-06-20 21:54:55 on dist-test-slave-kk5c"
server_key: "f14c6e98b5caf4712bb64e741a698125"
server_key_iv: "5526bf2d851ab51275cfe99ab05accde"
server_key_version: "encryptionkey@0"
I20260620 21:54:55.798475 22378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:55.829630 22378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:55.833459 22378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:55.833650 22378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:55.841403 22378 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:42339
I20260620 21:54:55.841487 22445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:42339 every 8 connection(s)
I20260620 21:54:55.842535 22378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:55.845703 22446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:55.847492 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 22378
I20260620 21:54:55.847671 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:55.847972 775 external_mini_cluster.cc:1428] Setting key db6644b29fe0de5b019c645e3043ab0f
I20260620 21:54:55.852488 22446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9: Bootstrap starting.
I20260620 21:54:55.855131 22446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:55.855999 22446 log.cc:824] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9: Log is configured to *not* fsync() on all Append() calls
Jun 20 21:54:55 dist-test-slave-kk5c krb5kdc[22365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:54:55.858520 22446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9: No bootstrap required, opened a new log
I20260620 21:54:55.863726 22446 raft_consensus.cc:348] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecbef5ba11c4d539609293b0b186ec9" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42339 } }
I20260620 21:54:55.864077 22446 raft_consensus.cc:374] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:55.864146 22446 raft_consensus.cc:729] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aecbef5ba11c4d539609293b0b186ec9, State: Initialized, Role: FOLLOWER
I20260620 21:54:55.864765 22446 consensus_queue.cc:260] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecbef5ba11c4d539609293b0b186ec9" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42339 } }
I20260620 21:54:55.864959 22446 raft_consensus.cc:388] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:55.865018 22446 raft_consensus.cc:482] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:55.865106 22446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:55.865537 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:55.849528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54445 (local address 127.0.193.254:42339)
0620 21:54:55.850010 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:55.850029 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:55.850066 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:54:55.850719 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:55.850736 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:55.851109 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:55.851541 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:55.852375 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:55.853727 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:55.854876 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:55.855609 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:55.855821 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:55.859235 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:55.859243 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:55.859263 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:55.859299 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:55.861700 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:55.862416 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:55.862423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:55.862431 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:55.862594 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:55.862984 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:55.862987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:55.862989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:55.863614 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:55.863834 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:55.864220 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 21:54:55.864466 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":144,"threads_started":1}
I20260620 21:54:55.866379 22446 raft_consensus.cc:504] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecbef5ba11c4d539609293b0b186ec9" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42339 } }
I20260620 21:54:55.866780 22446 leader_election.cc:302] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aecbef5ba11c4d539609293b0b186ec9; no voters:
I20260620 21:54:55.867127 22446 leader_election.cc:288] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:55.867280 22451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:55.867527 22451 raft_consensus.cc:686] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [term 1 LEADER]: Becoming Leader. State: Replica: aecbef5ba11c4d539609293b0b186ec9, State: Running, Role: LEADER
I20260620 21:54:55.867918 22451 consensus_queue.cc:237] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecbef5ba11c4d539609293b0b186ec9" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42339 } }
I20260620 21:54:55.868682 22446 sys_catalog.cc:564] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:55.869205 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:55.869992 22453 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aecbef5ba11c4d539609293b0b186ec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecbef5ba11c4d539609293b0b186ec9" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42339 } } }
I20260620 21:54:55.869943 22452 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aecbef5ba11c4d539609293b0b186ec9. Latest consensus state: current_term: 1 leader_uuid: "aecbef5ba11c4d539609293b0b186ec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecbef5ba11c4d539609293b0b186ec9" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42339 } } }
I20260620 21:54:55.870122 22453 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:55.870141 22452 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:55.870590 22461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:55.874796 22461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:55.881459 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:55.871741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40447 (local address 127.0.193.254:42339)
0620 21:54:55.871865 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:55.871869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:55.872209 (+ 340us) server_negotiation.cc:407] Connection header received
0620 21:54:55.872434 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:55.872437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:55.872479 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:55.872550 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:55.873490 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:55.874292 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:55.875487 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:55.875897 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:55.876017 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:55.877635 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:55.877639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:55.877641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:54:55.877670 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:55.879271 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:55.879908 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:55.879913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:55.879915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:55.880011 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:54:55.880354 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:54:55.880358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:54:55.880360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:54:55.880561 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:55.880726 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:55.881000 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 21:54:55.881102 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:54:55.881919 22461 catalog_manager.cc:1347] Generated new cluster ID: 4af1015d9cbd49abbd6b0b3576e5c4df
I20260620 21:54:55.881995 22461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:55.901767 22461 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:55.903424 22461 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:55.910375 22461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aecbef5ba11c4d539609293b0b186ec9: Generated new TSK 0
I20260620 21:54:55.911459 22461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:55.924208 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 22378
W20260620 21:54:55.930063 22363 connection.cc:535] client connection to 127.0.193.254:42339 recv error: Network error: failed to read from TLS socket (remote: 127.0.193.254:42339): Connection reset by peer (error 104)
2026-06-20T21:54:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1497 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:55 dist-test-slave-kk5c krb5kdc[22477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:55 dist-test-slave-kk5c krb5kdc[22477](info): set up 2 sockets
Jun 20 21:54:55 dist-test-slave-kk5c krb5kdc[22477](info): commencing operation
krb5kdc: starting...
W20260620 21:54:57.011639 775 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:57 dist-test-slave-kk5c krb5kdc[22477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:54:57.143687 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:40303
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:40303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:57.257838 22490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:57.258162 22490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:57.258204 22490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:57.262638 22490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:57.262732 22490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:57.262753 22490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:57.262770 22490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:57.262786 22490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:57.268661 22490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:40303
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:40303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:57.269882 22490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:57.270895 22490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:57.279877 22495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:57.279948 22496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:57.279979 22490 server_base.cc:1034] running on GCE node
W20260620 21:54:57.280223 22498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:57.280859 22490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:57.282116 22490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:57.283318 22490 hybrid_clock.cc:648] HybridClock initialized: now 1781992497283288 us; error 52 us; skew 500 ppm
Jun 20 21:54:57 dist-test-slave-kk5c krb5kdc[22477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:57.286134 22490 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:54:57.287113 22490 webserver.cc:460] Webserver started at http://127.0.193.254:35337/ using document root <none> and password file <none>
I20260620 21:54:57.287758 22490 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:57.287869 22490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:57.288108 22490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:57.290302 22490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "c8ea54b3c7f34e3c8b5bb53908aa1961"
format_stamp: "Formatted at 2026-06-20 21:54:57 on dist-test-slave-kk5c"
server_key: "ae818aa26b00d9012b8b1bc54ed8020d"
server_key_iv: "8376f1f737ebbf2e51ba9faaca6a6f6f"
server_key_version: "encryptionkey@0"
I20260620 21:54:57.290903 22490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "c8ea54b3c7f34e3c8b5bb53908aa1961"
format_stamp: "Formatted at 2026-06-20 21:54:57 on dist-test-slave-kk5c"
server_key: "ae818aa26b00d9012b8b1bc54ed8020d"
server_key_iv: "8376f1f737ebbf2e51ba9faaca6a6f6f"
server_key_version: "encryptionkey@0"
I20260620 21:54:57.295261 22490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 21:54:57.298482 22505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:57.300810 22490 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:54:57.301029 22490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "c8ea54b3c7f34e3c8b5bb53908aa1961"
format_stamp: "Formatted at 2026-06-20 21:54:57 on dist-test-slave-kk5c"
server_key: "ae818aa26b00d9012b8b1bc54ed8020d"
server_key_iv: "8376f1f737ebbf2e51ba9faaca6a6f6f"
server_key_version: "encryptionkey@0"
I20260620 21:54:57.301285 22490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:57.321475 22490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:57.324941 22490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:57.325177 22490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:57.332861 22490 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:40303
I20260620 21:54:57.332880 22557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:40303 every 8 connection(s)
I20260620 21:54:57.333892 22490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:57.337083 22558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:57.341144 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 22490
I20260620 21:54:57.341315 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:57.341617 775 external_mini_cluster.cc:1428] Setting key 84aba088412af32b01a131ef64f22827
I20260620 21:54:57.344125 22558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961: Bootstrap starting.
I20260620 21:54:57.346601 22558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:57.347520 22558 log.cc:824] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:57.349844 22558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961: No bootstrap required, opened a new log
I20260620 21:54:57.351670 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:57.343321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35687 (local address 127.0.193.254:40303)
0620 21:54:57.343987 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:57.344003 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:57.344040 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:54:57.344787 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:57.344805 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:57.345227 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:57.345578 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:57.346023 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:57.347102 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:57.348141 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:57.348642 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:57.348826 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:57.349753 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:57.349757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:57.349770 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:57.350208 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:57.350261 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:57.350286 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:57.350427 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:57.350609 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 21:54:57.350755 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":161,"threads_started":1}
I20260620 21:54:57.354972 22558 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ea54b3c7f34e3c8b5bb53908aa1961" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 40303 } }
I20260620 21:54:57.355338 22558 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:57.355406 22558 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ea54b3c7f34e3c8b5bb53908aa1961, State: Initialized, Role: FOLLOWER
I20260620 21:54:57.355506 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:57.355902 22558 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ea54b3c7f34e3c8b5bb53908aa1961" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 40303 } }
I20260620 21:54:57.356040 22558 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:57.356132 22558 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:57.356264 22558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:57.357463 22558 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ea54b3c7f34e3c8b5bb53908aa1961" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 40303 } }
I20260620 21:54:57.357901 22558 leader_election.cc:302] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8ea54b3c7f34e3c8b5bb53908aa1961; no voters:
I20260620 21:54:57.358327 22558 leader_election.cc:288] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:57.358417 22568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:57.358667 22568 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [term 1 LEADER]: Becoming Leader. State: Replica: c8ea54b3c7f34e3c8b5bb53908aa1961, State: Running, Role: LEADER
I20260620 21:54:57.359022 22568 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ea54b3c7f34e3c8b5bb53908aa1961" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 40303 } }
I20260620 21:54:57.359880 22558 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:57.360929 22569 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8ea54b3c7f34e3c8b5bb53908aa1961" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ea54b3c7f34e3c8b5bb53908aa1961" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 40303 } } }
I20260620 21:54:57.361073 22569 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:57.361552 22577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:57.361508 22570 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8ea54b3c7f34e3c8b5bb53908aa1961. Latest consensus state: current_term: 1 leader_uuid: "c8ea54b3c7f34e3c8b5bb53908aa1961" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ea54b3c7f34e3c8b5bb53908aa1961" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 40303 } } }
I20260620 21:54:57.361676 22570 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961 [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:57.364204 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:57.357950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58575 (local address 127.0.193.254:40303)
0620 21:54:57.358087 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:57.358121 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:57.358359 (+ 238us) server_negotiation.cc:407] Connection header received
0620 21:54:57.358589 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:57.358598 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:57.358645 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:57.358746 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:57.359226 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:57.359856 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:57.361205 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:54:57.361619 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:54:57.361732 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:54:57.363007 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:57.363011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:57.363014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:57.363058 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:57.363116 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:57.363133 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:57.363827 (+ 694us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:57.363914 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 21:54:57.363974 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:54:57.365628 22577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:57.372493 22577 catalog_manager.cc:1347] Generated new cluster ID: ead626e156424c228d4bb9b2dc058fc2
I20260620 21:54:57.372611 22577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:57.398115 22577 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:57.399304 22577 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:57.407755 22577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8ea54b3c7f34e3c8b5bb53908aa1961: Generated new TSK 0
I20260620 21:54:57.408571 22577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:57.414364 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 22490
W20260620 21:54:57.419934 22475 connection.cc:535] client connection to 127.0.193.254:40303 recv error: Network error: failed to read from TLS socket (remote: 127.0.193.254:40303): Connection reset by peer (error 104)
2026-06-20T21:54:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1489 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:57 dist-test-slave-kk5c krb5kdc[22589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:57 dist-test-slave-kk5c krb5kdc[22589](info): set up 2 sockets
Jun 20 21:54:57 dist-test-slave-kk5c krb5kdc[22589](info): commencing operation
krb5kdc: starting...
W20260620 21:54:58.503641 775 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:54:58 dist-test-slave-kk5c krb5kdc[22589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:54:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:54:58.648897 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:45161
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:45703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:45161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:54:58.771287 22602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:54:58.771603 22602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:54:58.771675 22602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:54:58.776373 22602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:54:58.776510 22602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:54:58.776533 22602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:54:58.776551 22602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:54:58.776566 22602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:54:58.782624 22602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:45703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:45161
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:45161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:54:58.783993 22602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:54:58.785172 22602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:54:58.794696 22608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:54:58.794780 22610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:58.794713 22602 server_base.cc:1034] running on GCE node
W20260620 21:54:58.794719 22607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:54:58.795652 22602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:54:58.796820 22602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:54:58.797996 22602 hybrid_clock.cc:648] HybridClock initialized: now 1781992498797978 us; error 35 us; skew 500 ppm
Jun 20 21:54:58 dist-test-slave-kk5c krb5kdc[22589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:54:58.800984 22602 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:54:58.802361 22602 webserver.cc:460] Webserver started at http://127.0.193.254:35671/ using document root <none> and password file <none>
I20260620 21:54:58.803128 22602 fs_manager.cc:359] Metadata directory not provided
I20260620 21:54:58.803261 22602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:54:58.803578 22602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:54:58.806363 22602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "86fe22d6f8cf41218a98edf9d7e8bb6d"
format_stamp: "Formatted at 2026-06-20 21:54:58 on dist-test-slave-kk5c"
server_key: "1a12c7c00ee73fdafaa5edd8842dfd06"
server_key_iv: "ec421e6f4e11e1937fdf2c5a7b7a0930"
server_key_version: "encryptionkey@0"
I20260620 21:54:58.807003 22602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "86fe22d6f8cf41218a98edf9d7e8bb6d"
format_stamp: "Formatted at 2026-06-20 21:54:58 on dist-test-slave-kk5c"
server_key: "1a12c7c00ee73fdafaa5edd8842dfd06"
server_key_iv: "ec421e6f4e11e1937fdf2c5a7b7a0930"
server_key_version: "encryptionkey@0"
I20260620 21:54:58.811748 22602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 21:54:58.814796 22617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:54:58.816421 22602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:54:58.816602 22602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "86fe22d6f8cf41218a98edf9d7e8bb6d"
format_stamp: "Formatted at 2026-06-20 21:54:58 on dist-test-slave-kk5c"
server_key: "1a12c7c00ee73fdafaa5edd8842dfd06"
server_key_iv: "ec421e6f4e11e1937fdf2c5a7b7a0930"
server_key_version: "encryptionkey@0"
I20260620 21:54:58.816744 22602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:54:58.822667 22602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:54:58.826406 22602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:54:58.826668 22602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:54:58.834425 22602 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:45161
I20260620 21:54:58.834640 22669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:45161 every 8 connection(s)
I20260620 21:54:58.835532 22602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:54:58.836620 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 22602
I20260620 21:54:58.836853 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:54:58.837150 775 external_mini_cluster.cc:1428] Setting key 3038edea24cd15f0d08fc7f2ae07d72c
I20260620 21:54:58.840552 22670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:54:58.843406 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:58.838603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54753 (local address 127.0.193.254:45161)
0620 21:54:58.839419 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:58.839436 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:58.839472 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:54:58.840336 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:58.840354 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:58.840685 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:58.841315 (+ 630us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:58.842131 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:58.842135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:58.842148 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:58.842159 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:58.842217 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:58.842238 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:58.842392 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:58.842461 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 21:54:58.842628 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":256,"threads_started":1}
I20260620 21:54:58.847182 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:54:58.847889 22670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d: Bootstrap starting.
I20260620 21:54:58.850903 22670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d: Neither blocks nor log segments found. Creating new log.
I20260620 21:54:58.851606 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:54:58.849591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55189 (local address 127.0.193.254:45161)
0620 21:54:58.849702 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 21:54:58.849706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:54:58.849811 (+ 105us) server_negotiation.cc:407] Connection header received
0620 21:54:58.849940 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:54:58.849942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:54:58.849975 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:54:58.850058 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:54:58.850937 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:54:58.850940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:54:58.850942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:54:58.850953 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:54:58.850981 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:54:58.850991 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:54:58.851168 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 21:54:58.851367 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 21:54:58.851443 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 21:54:58.851984 22670 log.cc:824] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d: Log is configured to *not* fsync() on all Append() calls
I20260620 21:54:58.854282 22670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d: No bootstrap required, opened a new log
I20260620 21:54:58.858590 22670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86fe22d6f8cf41218a98edf9d7e8bb6d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45161 } }
I20260620 21:54:58.858906 22670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:54:58.858986 22670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86fe22d6f8cf41218a98edf9d7e8bb6d, State: Initialized, Role: FOLLOWER
I20260620 21:54:58.859493 22670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86fe22d6f8cf41218a98edf9d7e8bb6d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45161 } }
I20260620 21:54:58.859671 22670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:54:58.859758 22670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:54:58.859863 22670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:54:58.860877 22670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86fe22d6f8cf41218a98edf9d7e8bb6d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45161 } }
I20260620 21:54:58.861364 22670 leader_election.cc:302] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86fe22d6f8cf41218a98edf9d7e8bb6d; no voters:
I20260620 21:54:58.861770 22670 leader_election.cc:288] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:54:58.861930 22680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:54:58.862195 22680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [term 1 LEADER]: Becoming Leader. State: Replica: 86fe22d6f8cf41218a98edf9d7e8bb6d, State: Running, Role: LEADER
I20260620 21:54:58.862563 22680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86fe22d6f8cf41218a98edf9d7e8bb6d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45161 } }
I20260620 21:54:58.863163 22670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:54:58.864331 22682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86fe22d6f8cf41218a98edf9d7e8bb6d. Latest consensus state: current_term: 1 leader_uuid: "86fe22d6f8cf41218a98edf9d7e8bb6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86fe22d6f8cf41218a98edf9d7e8bb6d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45161 } } }
I20260620 21:54:58.864480 22682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:58.864877 22689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:54:58.865206 22681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86fe22d6f8cf41218a98edf9d7e8bb6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86fe22d6f8cf41218a98edf9d7e8bb6d" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 45161 } } }
I20260620 21:54:58.865307 22681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d [sys.catalog]: This master's current role is: LEADER
I20260620 21:54:58.868618 22689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:54:58.875276 22689 catalog_manager.cc:1347] Generated new cluster ID: 52239fe8260e42d1b732de61f14e2f3f
I20260620 21:54:58.875375 22689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:54:58.883715 22689 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:54:58.884128 22689 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:54:58.896953 22689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86fe22d6f8cf41218a98edf9d7e8bb6d: Generated new TSK 0
I20260620 21:54:58.897943 22689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:54:58.910723 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 22602
2026-06-20T21:54:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1495 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:54:58 dist-test-slave-kk5c krb5kdc[22701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:54:58 dist-test-slave-kk5c krb5kdc[22701](info): set up 2 sockets
Jun 20 21:54:58 dist-test-slave-kk5c krb5kdc[22701](info): commencing operation
krb5kdc: starting...
W20260620 21:54:59.959564 775 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:55:00 dist-test-slave-kk5c krb5kdc[22701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:55:00.091267 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:38797
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:45467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:38797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:00.205907 22714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:00.206259 22714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:00.206346 22714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:00.210564 22714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:00.210675 22714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:00.210716 22714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:00.210772 22714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:00.210829 22714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:00.216398 22714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:45467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:38797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:38797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:00.217628 22714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:00.218669 22714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:00.227398 22722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:00.227406 22719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:00.227533 22714 server_base.cc:1034] running on GCE node
W20260620 21:55:00.227427 22720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:00.228235 22714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:00.229300 22714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:00.230504 22714 hybrid_clock.cc:648] HybridClock initialized: now 1781992500230468 us; error 46 us; skew 500 ppm
Jun 20 21:55:00 dist-test-slave-kk5c krb5kdc[22701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:00.233819 22714 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:55:00.234820 22714 webserver.cc:460] Webserver started at http://127.0.193.254:45495/ using document root <none> and password file <none>
I20260620 21:55:00.235477 22714 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:00.235571 22714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:00.235805 22714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:00.237895 22714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "90162e5abe014b50a9062de0b2b161cd"
format_stamp: "Formatted at 2026-06-20 21:55:00 on dist-test-slave-kk5c"
server_key: "e7943994223aab88ca69785ce2dfeb23"
server_key_iv: "a4b7fd5b8920e193360b53f0f8feb100"
server_key_version: "encryptionkey@0"
I20260620 21:55:00.238476 22714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "90162e5abe014b50a9062de0b2b161cd"
format_stamp: "Formatted at 2026-06-20 21:55:00 on dist-test-slave-kk5c"
server_key: "e7943994223aab88ca69785ce2dfeb23"
server_key_iv: "a4b7fd5b8920e193360b53f0f8feb100"
server_key_version: "encryptionkey@0"
I20260620 21:55:00.242359 22714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 21:55:00.245119 22729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:00.246444 22714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 21:55:00.246615 22714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "90162e5abe014b50a9062de0b2b161cd"
format_stamp: "Formatted at 2026-06-20 21:55:00 on dist-test-slave-kk5c"
server_key: "e7943994223aab88ca69785ce2dfeb23"
server_key_iv: "a4b7fd5b8920e193360b53f0f8feb100"
server_key_version: "encryptionkey@0"
I20260620 21:55:00.246738 22714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:00.265925 22714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:00.269393 22714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:00.269615 22714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:00.277515 22714 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:38797
I20260620 21:55:00.277539 22781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:38797 every 8 connection(s)
I20260620 21:55:00.278527 22714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:00.281332 22782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:00.287029 22782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd: Bootstrap starting.
I20260620 21:55:00.288468 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 22714
I20260620 21:55:00.288614 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:55:00.288872 775 external_mini_cluster.cc:1428] Setting key cdbe13be081081a2e0435276c8f5c109
I20260620 21:55:00.290107 22782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:00.290987 22782 log.cc:824] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:00.293138 22782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd: No bootstrap required, opened a new log
I20260620 21:55:00.297242 22782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90162e5abe014b50a9062de0b2b161cd" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38797 } }
I20260620 21:55:00.297501 22782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:00.297590 22782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90162e5abe014b50a9062de0b2b161cd, State: Initialized, Role: FOLLOWER
Jun 20 21:55:00 dist-test-slave-kk5c krb5kdc[22701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:55:00.298074 22782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90162e5abe014b50a9062de0b2b161cd" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38797 } }
I20260620 21:55:00.298269 22782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:00.298354 22782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:00.298450 22782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:00.299484 22782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90162e5abe014b50a9062de0b2b161cd" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38797 } }
I20260620 21:55:00.299842 22782 leader_election.cc:302] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90162e5abe014b50a9062de0b2b161cd; no voters:
I20260620 21:55:00.300222 22782 leader_election.cc:288] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:00.300477 22787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:00.300736 22787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [term 1 LEADER]: Becoming Leader. State: Replica: 90162e5abe014b50a9062de0b2b161cd, State: Running, Role: LEADER
I20260620 21:55:00.301090 22787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90162e5abe014b50a9062de0b2b161cd" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38797 } }
I20260620 21:55:00.301682 22782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:00.302848 22788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90162e5abe014b50a9062de0b2b161cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90162e5abe014b50a9062de0b2b161cd" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38797 } } }
I20260620 21:55:00.303028 22788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:00.303350 22788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90162e5abe014b50a9062de0b2b161cd. Latest consensus state: current_term: 1 leader_uuid: "90162e5abe014b50a9062de0b2b161cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90162e5abe014b50a9062de0b2b161cd" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 38797 } } }
I20260620 21:55:00.303496 22788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:00.305151 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:00.290154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56973 (local address 127.0.193.254:38797)
0620 21:55:00.290635 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:00.290650 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:00.290687 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:55:00.291568 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:00.291596 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:00.292059 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:00.292411 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:00.293252 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:00.294290 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:00.295179 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:00.295710 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:00.295891 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:00.298791 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:00.298796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:00.298811 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:00.298840 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:00.301068 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:00.301637 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:00.301643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:00.301649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:00.301798 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:00.302120 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:00.302123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:00.302125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:00.302681 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:00.303687 (+ 1006us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:00.303851 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 21:55:00.304256 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":164,"threads_started":1}
W20260620 21:55:00.307178 22802 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:55:00.307271 22802 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:55:00.307384 22796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:00.308228 22796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:00.308281 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:55:00.314831 22796 catalog_manager.cc:1347] Generated new cluster ID: 8d00c94969334ddcac6ad8299a9fce8f
I20260620 21:55:00.314908 22796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:00.319483 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:00.310556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38089 (local address 127.0.193.254:38797)
0620 21:55:00.310675 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:00.310679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:00.310927 (+ 248us) server_negotiation.cc:407] Connection header received
0620 21:55:00.311108 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:00.311113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:00.311165 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:00.311275 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:00.311941 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:00.312542 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:00.313426 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:00.313676 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:00.313788 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:00.315671 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:00.315674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:00.315677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:00.315706 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:00.317412 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:00.317892 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:00.317896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:00.317897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:00.317973 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:00.318222 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:00.318225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:00.318226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:00.318411 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:00.318516 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:00.319207 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 21:55:00.319329 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:55:00.327348 22796 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:00.328631 22796 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:00.340793 22796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90162e5abe014b50a9062de0b2b161cd: Generated new TSK 0
I20260620 21:55:00.341692 22796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:55:00.345156 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 22714
2026-06-20T21:55:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:00 dist-test-slave-kk5c krb5kdc[22813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:00 dist-test-slave-kk5c krb5kdc[22813](info): set up 2 sockets
Jun 20 21:55:00 dist-test-slave-kk5c krb5kdc[22813](info): commencing operation
krb5kdc: starting...
W20260620 21:55:01.423581 775 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:55:01 dist-test-slave-kk5c krb5kdc[22813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:55:01.551121 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:37095
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:37095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:01.673269 22826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:01.673650 22826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:01.673754 22826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:01.677948 22826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:01.678058 22826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:01.678097 22826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:01.678154 22826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:01.678194 22826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:01.684047 22826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:37095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:37095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:01.685199 22826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:01.686267 22826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:01.696178 22834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:01.696178 22831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:01.696317 22832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:01.696419 22826 server_base.cc:1034] running on GCE node
I20260620 21:55:01.697556 22826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:01.699126 22826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:01.700562 22826 hybrid_clock.cc:648] HybridClock initialized: now 1781992501700483 us; error 96 us; skew 500 ppm
Jun 20 21:55:01 dist-test-slave-kk5c krb5kdc[22813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:01.703783 22826 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:55:01.704789 22826 webserver.cc:460] Webserver started at http://127.0.193.254:45191/ using document root <none> and password file <none>
I20260620 21:55:01.705385 22826 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:01.705490 22826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:01.705739 22826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:01.708060 22826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "5c449e849c594131a6eb18546bfd6baa"
format_stamp: "Formatted at 2026-06-20 21:55:01 on dist-test-slave-kk5c"
server_key: "a9959422c21170e52e534589c9efd8a0"
server_key_iv: "a12a9b14d247a43a7613c59360211814"
server_key_version: "encryptionkey@0"
I20260620 21:55:01.708674 22826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "5c449e849c594131a6eb18546bfd6baa"
format_stamp: "Formatted at 2026-06-20 21:55:01 on dist-test-slave-kk5c"
server_key: "a9959422c21170e52e534589c9efd8a0"
server_key_iv: "a12a9b14d247a43a7613c59360211814"
server_key_version: "encryptionkey@0"
I20260620 21:55:01.713241 22826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:55:01.716114 22841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:01.717556 22826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 21:55:01.717727 22826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "5c449e849c594131a6eb18546bfd6baa"
format_stamp: "Formatted at 2026-06-20 21:55:01 on dist-test-slave-kk5c"
server_key: "a9959422c21170e52e534589c9efd8a0"
server_key_iv: "a12a9b14d247a43a7613c59360211814"
server_key_version: "encryptionkey@0"
I20260620 21:55:01.717851 22826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:01.738715 22826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:01.742237 22826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:01.742482 22826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:01.750403 22826 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:37095
I20260620 21:55:01.750476 22893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:37095 every 8 connection(s)
I20260620 21:55:01.751509 22826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:01.754652 22894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:01.758472 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 22826
I20260620 21:55:01.758637 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:55:01.758953 775 external_mini_cluster.cc:1428] Setting key 83bfbe08e83b5acf04796fa3e3c5f28a
I20260620 21:55:01.761284 22894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa: Bootstrap starting.
I20260620 21:55:01.764043 22894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:01.765103 22894 log.cc:824] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:01.767727 22894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa: No bootstrap required, opened a new log
I20260620 21:55:01.769322 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:01.760540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48493 (local address 127.0.193.254:37095)
0620 21:55:01.760997 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:01.761014 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:01.761051 (+ 37us) server_negotiation.cc:407] Connection header received
0620 21:55:01.761794 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:01.761814 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:01.762193 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:01.762493 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:01.762995 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:01.764091 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:01.765523 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:01.766235 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:01.766362 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:01.767350 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:01.767354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:01.767369 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:01.767844 (+ 475us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:01.767922 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:01.767950 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:01.768041 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:01.768241 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 21:55:01.768566 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":148,"threads_started":1}
I20260620 21:55:01.772007 22894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c449e849c594131a6eb18546bfd6baa" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 37095 } }
I20260620 21:55:01.772264 22894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:01.772301 22894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c449e849c594131a6eb18546bfd6baa, State: Initialized, Role: FOLLOWER
I20260620 21:55:01.772675 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:55:01.772876 22894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c449e849c594131a6eb18546bfd6baa" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 37095 } }
I20260620 21:55:01.773066 22894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:01.773151 22894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:01.773294 22894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:01.774477 22894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c449e849c594131a6eb18546bfd6baa" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 37095 } }
I20260620 21:55:01.774860 22894 leader_election.cc:302] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c449e849c594131a6eb18546bfd6baa; no voters:
I20260620 21:55:01.775274 22894 leader_election.cc:288] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:01.775336 22903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:01.775612 22903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [term 1 LEADER]: Becoming Leader. State: Replica: 5c449e849c594131a6eb18546bfd6baa, State: Running, Role: LEADER
I20260620 21:55:01.775969 22903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c449e849c594131a6eb18546bfd6baa" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 37095 } }
I20260620 21:55:01.776801 22894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:01.777274 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c449e849c594131a6eb18546bfd6baa. Latest consensus state: current_term: 1 leader_uuid: "5c449e849c594131a6eb18546bfd6baa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c449e849c594131a6eb18546bfd6baa" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 37095 } } }
I20260620 21:55:01.777458 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:01.777736 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c449e849c594131a6eb18546bfd6baa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c449e849c594131a6eb18546bfd6baa" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 37095 } } }
I20260620 21:55:01.777904 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:01.780033 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:01.775286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46987 (local address 127.0.193.254:37095)
0620 21:55:01.775422 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:01.775427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:01.775643 (+ 216us) server_negotiation.cc:407] Connection header received
0620 21:55:01.775825 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:01.775828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:01.775883 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:01.775972 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:01.776265 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:01.776791 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:01.777930 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:01.778256 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:01.778335 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:01.779424 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:01.779428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:01.779431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:01.779444 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:01.779472 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:01.779487 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:01.779538 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:01.779842 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 21:55:01.779897 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 21:55:01.782454 22920 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:55:01.782545 22920 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:55:01.782655 22908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:01.783617 22908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:01.789721 22908 catalog_manager.cc:1347] Generated new cluster ID: f526fd3a81104707ab2e5cf1fd280056
I20260620 21:55:01.789803 22908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:01.808360 22908 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:01.809701 22908 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:01.816388 22908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c449e849c594131a6eb18546bfd6baa: Generated new TSK 0
I20260620 21:55:01.817341 22908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:55:01.826095 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 22826
2026-06-20T21:55:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:01 dist-test-slave-kk5c krb5kdc[22925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:01 dist-test-slave-kk5c krb5kdc[22925](info): set up 2 sockets
Jun 20 21:55:01 dist-test-slave-kk5c krb5kdc[22925](info): commencing operation
krb5kdc: starting...
W20260620 21:55:02.903605 775 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:55:02 dist-test-slave-kk5c krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:55:03.040737 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:42775
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:35219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:42775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:03.155548 22938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:03.155843 22938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:03.155884 22938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:03.160179 22938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:03.160259 22938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:03.160279 22938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:03.160296 22938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:03.160311 22938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:03.165861 22938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:35219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:42775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:42775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:03.167155 22938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:03.168342 22938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:03.177162 22943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:03.177156 22944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:03.177366 22946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:03.177454 22938 server_base.cc:1034] running on GCE node
I20260620 21:55:03.178177 22938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:03.179445 22938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:03.180636 22938 hybrid_clock.cc:648] HybridClock initialized: now 1781992503180604 us; error 50 us; skew 500 ppm
Jun 20 21:55:03 dist-test-slave-kk5c krb5kdc[22925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:03.187032 22938 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:55:03.188179 22938 webserver.cc:460] Webserver started at http://127.0.193.254:34121/ using document root <none> and password file <none>
I20260620 21:55:03.188781 22938 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:03.188874 22938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:03.189118 22938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:03.191282 22938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "102d92a2f15a48f1bd627c1bc3722050"
format_stamp: "Formatted at 2026-06-20 21:55:03 on dist-test-slave-kk5c"
server_key: "ab193a42768873d8800970160f588d2e"
server_key_iv: "9215d28aec25cdcbd8296efbc90f9799"
server_key_version: "encryptionkey@0"
I20260620 21:55:03.191943 22938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "102d92a2f15a48f1bd627c1bc3722050"
format_stamp: "Formatted at 2026-06-20 21:55:03 on dist-test-slave-kk5c"
server_key: "ab193a42768873d8800970160f588d2e"
server_key_iv: "9215d28aec25cdcbd8296efbc90f9799"
server_key_version: "encryptionkey@0"
I20260620 21:55:03.195853 22938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 21:55:03.198486 22953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:03.199949 22938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:03.200105 22938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "102d92a2f15a48f1bd627c1bc3722050"
format_stamp: "Formatted at 2026-06-20 21:55:03 on dist-test-slave-kk5c"
server_key: "ab193a42768873d8800970160f588d2e"
server_key_iv: "9215d28aec25cdcbd8296efbc90f9799"
server_key_version: "encryptionkey@0"
I20260620 21:55:03.200235 22938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:03.205063 22938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:03.208868 22938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:03.209112 22938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:03.217047 22938 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:42775
I20260620 21:55:03.217073 23005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:42775 every 8 connection(s)
I20260620 21:55:03.218135 22938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:03.220909 23006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:03.226336 23006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050: Bootstrap starting.
I20260620 21:55:03.227643 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 22938
I20260620 21:55:03.227798 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:55:03.228128 775 external_mini_cluster.cc:1428] Setting key 813310685ca259f2aa235a3c2572a704
I20260620 21:55:03.229130 23006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:03.229867 23006 log.cc:824] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:03.232060 23006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050: No bootstrap required, opened a new log
I20260620 21:55:03.233839 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:03.229420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50233 (local address 127.0.193.254:42775)
0620 21:55:03.229922 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:03.229939 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:03.229975 (+ 36us) server_negotiation.cc:407] Connection header received
0620 21:55:03.230901 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:03.230921 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:03.231295 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:03.231603 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:03.232484 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:03.232488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:03.232500 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:03.232511 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:03.232565 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:03.232586 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:03.232679 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:03.232836 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 21:55:03.233068 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":154,"threads_started":1}
I20260620 21:55:03.235958 23006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102d92a2f15a48f1bd627c1bc3722050" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42775 } }
I20260620 21:55:03.236235 23006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 21:55:03.236295 23006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102d92a2f15a48f1bd627c1bc3722050, State: Initialized, Role: FOLLOWER
I20260620 21:55:03.236702 23006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102d92a2f15a48f1bd627c1bc3722050" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42775 } }
I20260620 21:55:03.236860 23006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 21:55:03.236933 23006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 21:55:03.236956 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:55:03.237035 23006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:03.238293 23006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102d92a2f15a48f1bd627c1bc3722050" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42775 } }
I20260620 21:55:03.238698 23006 leader_election.cc:302] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 102d92a2f15a48f1bd627c1bc3722050; no voters:
I20260620 21:55:03.239035 23006 leader_election.cc:288] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 21:55:03.239157 23015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:03.239423 23015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [term 1 LEADER]: Becoming Leader. State: Replica: 102d92a2f15a48f1bd627c1bc3722050, State: Running, Role: LEADER
I20260620 21:55:03.239748 23015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102d92a2f15a48f1bd627c1bc3722050" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42775 } }
I20260620 21:55:03.240496 23006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:03.241221 23017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 102d92a2f15a48f1bd627c1bc3722050. Latest consensus state: current_term: 1 leader_uuid: "102d92a2f15a48f1bd627c1bc3722050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102d92a2f15a48f1bd627c1bc3722050" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42775 } } }
I20260620 21:55:03.241364 23017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:03.241741 23016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "102d92a2f15a48f1bd627c1bc3722050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102d92a2f15a48f1bd627c1bc3722050" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 42775 } } }
I20260620 21:55:03.241842 23016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:03.241895 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:03.239329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53061 (local address 127.0.193.254:42775)
0620 21:55:03.239515 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:03.239520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:03.239840 (+ 320us) server_negotiation.cc:407] Connection header received
0620 21:55:03.240011 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:03.240014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:03.240057 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:03.240156 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:03.241101 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:03.241105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:03.241107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 21:55:03.241122 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:03.241159 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 21:55:03.241176 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:03.241261 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:03.241718 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 21:55:03.241767 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 21:55:03.245450 23031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:55:03.245556 23031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:55:03.245635 23023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:03.246563 23023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:03.252620 23023 catalog_manager.cc:1347] Generated new cluster ID: 5ee23916d1ef4dcdbfd62fb7fbdf72a0
I20260620 21:55:03.252708 23023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:03.264066 23023 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:03.264443 23023 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:03.272306 23023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 102d92a2f15a48f1bd627c1bc3722050: Generated new TSK 0
I20260620 21:55:03.273106 23023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 21:55:03.280120 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 22938
2026-06-20T21:55:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1453 ms)
[----------] 18 tests from AuthTokenIssuingTest (26511 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:03 dist-test-slave-kk5c krb5kdc[23037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:03 dist-test-slave-kk5c krb5kdc[23037](info): set up 2 sockets
Jun 20 21:55:03 dist-test-slave-kk5c krb5kdc[23037](info): commencing operation
krb5kdc: starting...
W20260620 21:55:04.352001 775 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:55:04 dist-test-slave-kk5c krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:55:04.482360 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36395
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:36395,127.0.193.253:41889,127.0.193.252:44517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:04.597183 23050 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:55:04.597604 23050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:04.597682 23050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:04.597761 23050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:04.602156 23050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:04.602264 23050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:04.602301 23050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:04.602347 23050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:04.602386 23050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:04.608155 23050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:36395,127.0.193.253:41889,127.0.193.252:44517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:04.609390 23050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:04.610491 23050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:04.620048 23056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:04.620105 23058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:04.620072 23055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:04.620354 23050 server_base.cc:1034] running on GCE node
I20260620 21:55:04.621246 23050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:04.622578 23050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:04.623786 23050 hybrid_clock.cc:648] HybridClock initialized: now 1781992504623775 us; error 71 us; skew 500 ppm
Jun 20 21:55:04 dist-test-slave-kk5c krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:04.627563 23050 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:55:04.628713 23050 webserver.cc:460] Webserver started at http://127.0.193.254:44455/ using document root <none> and password file <none>
I20260620 21:55:04.629361 23050 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:04.629472 23050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:04.629736 23050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:04.632316 23050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "d67e8eed3e154e5ca95a6435386bf652"
format_stamp: "Formatted at 2026-06-20 21:55:04 on dist-test-slave-kk5c"
server_key: "07ab91bf4dcbeb732d1fa2116fd07659"
server_key_iv: "fb6b08d508cbf0b9aa6c42c9d07d7539"
server_key_version: "encryptionkey@0"
I20260620 21:55:04.632927 23050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "d67e8eed3e154e5ca95a6435386bf652"
format_stamp: "Formatted at 2026-06-20 21:55:04 on dist-test-slave-kk5c"
server_key: "07ab91bf4dcbeb732d1fa2116fd07659"
server_key_iv: "fb6b08d508cbf0b9aa6c42c9d07d7539"
server_key_version: "encryptionkey@0"
I20260620 21:55:04.637050 23050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 21:55:04.639966 23065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:04.641351 23050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:04.641559 23050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "d67e8eed3e154e5ca95a6435386bf652"
format_stamp: "Formatted at 2026-06-20 21:55:04 on dist-test-slave-kk5c"
server_key: "07ab91bf4dcbeb732d1fa2116fd07659"
server_key_iv: "fb6b08d508cbf0b9aa6c42c9d07d7539"
server_key_version: "encryptionkey@0"
I20260620 21:55:04.641703 23050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:04.667593 23050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:04.670985 23050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:04.671236 23050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:04.679225 23050 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:36395
I20260620 21:55:04.679522 23117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:36395 every 8 connection(s)
I20260620 21:55:04.680423 23050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:04.683410 23118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:04.687387 23118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:55:04.690229 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 23050
I20260620 21:55:04.690378 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:55:04.690730 775 external_mini_cluster.cc:1428] Setting key 2d81bb9567e1c1590735883b45fa5c73
Jun 20 21:55:04 dist-test-slave-kk5c krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.193.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.253@KRBTEST.COM" created.
I20260620 21:55:04.705246 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.689826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.0.193.254:36395)
0620 21:55:04.690469 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:04.690489 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:04.690766 (+ 277us) server_negotiation.cc:407] Connection header received
0620 21:55:04.691885 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:04.691894 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:04.692135 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:04.692367 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:04.693595 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.694617 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.696024 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.696899 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.697129 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:04.699940 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:04.699946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:04.699958 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:04.699979 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:04.701852 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:04.702374 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:04.702379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:04.702381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:04.702521 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:04.702972 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:04.702976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:04.702979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:04.703587 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:04.703729 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:04.704161 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 21:55:04.704495 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":453,"thread_start_us":150,"threads_started":1}
I20260620 21:55:04.705250 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.689840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36395 (local address 127.0.0.1:48022)
0620 21:55:04.690581 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.690635 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:04.691560 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:04.692490 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:04.692499 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:04.692995 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:04.693413 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:04.693434 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.694762 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.694765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:04.695872 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:04.695882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.697098 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.697101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:04.697204 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:04.697947 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:04.697964 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:04.699717 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:04.701958 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:04.701966 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:04.701975 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:04.702220 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:04.702654 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:04.702660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:04.702665 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:04.702826 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:04.703726 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:04.703731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:04.703860 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 21:55:04.704032 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 21:55:04.704391 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":74,"spinlock_wait_cycles":54016,"thread_start_us":159,"threads_started":1}
I20260620 21:55:04.707695 23118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:55:04.708501 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.708225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:41889 (local address 127.0.0.1:53378)
0620 21:55:04.708361 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.708439 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 21:55:04.708704 23067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.193.253:41889: connect: Connection refused (error 111)
I20260620 21:55:04.709544 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.709341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:41889 (local address 127.0.0.1:53390)
0620 21:55:04.709445 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.709494 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 21:55:04.709702 23118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.253:41889: Network error: Client connection negotiation failed: client connection to 127.0.193.253:41889: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.253.keytab.
Entry for principal kudu/127.0.193.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.253.keytab.
I20260620 21:55:04.722504 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.253:41889
--webserver_interface=127.0.193.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.253.keytab
--principal=kudu/127.0.193.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:36395,127.0.193.253:41889,127.0.193.252:44517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:55:04.758944 23118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } attempt: 1
I20260620 21:55:04.760080 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.759761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:41889 (local address 127.0.0.1:53394)
0620 21:55:04.759935 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.760004 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 21:55:04.760828 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.760609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:41889 (local address 127.0.0.1:53410)
0620 21:55:04.760717 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.760770 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 21:55:04.760962 23118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.253:41889: Network error: Client connection negotiation failed: client connection to 127.0.193.253:41889: connect: Connection refused (error 111)
I20260620 21:55:04.829138 23118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } attempt: 2
I20260620 21:55:04.830035 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.829736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:41889 (local address 127.0.0.1:53424)
0620 21:55:04.829892 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.829956 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 21:55:04.830760 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.830541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:41889 (local address 127.0.0.1:53428)
0620 21:55:04.830651 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.830703 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 21:55:04.830892 23118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.253:41889: Network error: Client connection negotiation failed: client connection to 127.0.193.253:41889: connect: Connection refused (error 111)
W20260620 21:55:04.845407 23125 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:55:04.845777 23125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:04.845834 23125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:04.845892 23125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:04.850148 23125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:04.850281 23125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:04.850324 23125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:04.850354 23125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:04.850458 23125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:04.856719 23125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:36395,127.0.193.253:41889,127.0.193.252:44517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.253.keytab
--principal=kudu/127.0.193.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.253:41889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.193.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:04.857867 23125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:04.858904 23125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:04.868467 23133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:04.868577 23130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:04.868520 23131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:04.868791 23125 server_base.cc:1034] running on GCE node
I20260620 21:55:04.869774 23125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:04.871083 23125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:04.872293 23125 hybrid_clock.cc:648] HybridClock initialized: now 1781992504872278 us; error 34 us; skew 500 ppm
Jun 20 21:55:04 dist-test-slave-kk5c krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:04.875768 23125 init.cc:377] Logged in from keytab as kudu/127.0.193.253@KRBTEST.COM (short username kudu)
I20260620 21:55:04.876873 23125 webserver.cc:460] Webserver started at http://127.0.193.253:40059/ using document root <none> and password file <none>
I20260620 21:55:04.877509 23125 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:04.877609 23125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:04.877852 23125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:04.880748 23125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/data/instance:
uuid: "fd49d46edb084ae68593d5d89c7666fd"
format_stamp: "Formatted at 2026-06-20 21:55:04 on dist-test-slave-kk5c"
server_key: "18fcb082a115d860d4e04c67320859ca"
server_key_iv: "6b5d0187c96e43b429bd2a2f302cfc59"
server_key_version: "encryptionkey@0"
I20260620 21:55:04.881587 23125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/wal/instance:
uuid: "fd49d46edb084ae68593d5d89c7666fd"
format_stamp: "Formatted at 2026-06-20 21:55:04 on dist-test-slave-kk5c"
server_key: "18fcb082a115d860d4e04c67320859ca"
server_key_iv: "6b5d0187c96e43b429bd2a2f302cfc59"
server_key_version: "encryptionkey@0"
I20260620 21:55:04.886138 23125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:55:04.889302 23140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:04.890838 23125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:55:04.891011 23125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/wal
uuid: "fd49d46edb084ae68593d5d89c7666fd"
format_stamp: "Formatted at 2026-06-20 21:55:04 on dist-test-slave-kk5c"
server_key: "18fcb082a115d860d4e04c67320859ca"
server_key_iv: "6b5d0187c96e43b429bd2a2f302cfc59"
server_key_version: "encryptionkey@0"
I20260620 21:55:04.891160 23125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:04.903877 23125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:04.907305 23125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:04.907517 23125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:04.915382 23125 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.253:41889
I20260620 21:55:04.915766 23192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.253:41889 every 8 connection(s)
I20260620 21:55:04.916461 23125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/data/info.pb
I20260620 21:55:04.919564 23193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:04.919936 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 23125
I20260620 21:55:04.920099 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-1/wal/instance
I20260620 21:55:04.920373 775 external_mini_cluster.cc:1428] Setting key 32d69aa88b3ff24afeca664d182273e0
I20260620 21:55:04.922083 23118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } attempt: 3
I20260620 21:55:04.925088 23193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.193.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.252@KRBTEST.COM" created.
Jun 20 21:55:04 dist-test-slave-kk5c krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for kudu/127.0.193.253@KRBTEST.COM
Jun 20 21:55:04 dist-test-slave-kk5c krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.253@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:55:04.937976 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.922678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:41889 (local address 127.0.0.1:53432)
0620 21:55:04.922966 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.922979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:04.923167 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:04.925817 (+ 2650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:04.925824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:04.926067 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:04.926248 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:04.926259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.928039 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.928044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:04.928985 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:04.928997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.930008 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.930013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:04.930223 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:04.930751 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:04.930767 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:04.933031 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:04.935792 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:04.935796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:04.935800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:04.936011 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:04.936411 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:04.936417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:04.936421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:04.936532 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:04.937318 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:04.937321 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:04.937432 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 21:55:04.937546 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 21:55:04.937713 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260620 21:55:04.938355 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.927163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48024 (local address 127.0.193.254:36395)
0620 21:55:04.927310 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:04.927315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:04.927692 (+ 377us) server_negotiation.cc:407] Connection header received
0620 21:55:04.927917 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:04.927922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:04.927980 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:04.928104 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:04.928888 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.929635 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.930935 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.931315 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.931440 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:04.934244 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:04.934248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:04.934250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:04.934304 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:04.935721 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:04.936286 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:04.936289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:04.936293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:04.936376 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:04.936713 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:04.936718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:04.936723 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:04.936988 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:04.937102 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:04.938092 (+ 990us) server_negotiation.cc:299] Negotiation successful
0620 21:55:04.938216 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":74}
I20260620 21:55:04.938891 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.927130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36395 (local address 127.0.0.1:48024)
0620 21:55:04.927578 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.927626 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:04.927794 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:04.928284 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:04.928289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:04.928633 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:04.928750 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:04.928755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.929797 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.929800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:04.930805 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:04.930819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.931486 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.931490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:04.931604 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:04.932143 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:04.932164 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:04.934101 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:04.935851 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:04.935861 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:04.935875 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:04.936180 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:04.936451 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:04.936455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:04.936458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:04.936574 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:04.937103 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:04.937110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:04.937241 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 21:55:04.937421 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 21:55:04.937800 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":157,"threads_started":1}
I20260620 21:55:04.939580 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.923492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53432 (local address 127.0.193.253:41889)
0620 21:55:04.924010 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:04.924029 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:04.924072 (+ 43us) server_negotiation.cc:407] Connection header received
0620 21:55:04.924934 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:04.924995 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:04.925436 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:04.925819 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:04.926417 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.927850 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.929124 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.929870 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.929989 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:04.933189 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:04.933200 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:04.933211 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:04.933233 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:04.935671 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:04.936124 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:04.936130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:04.936134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:04.936274 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:04.936649 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:04.936653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:04.936656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:04.937193 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:04.937320 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:04.938823 (+ 1503us) server_negotiation.cc:299] Negotiation successful
0620 21:55:04.938991 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":155,"threads_started":1}
I20260620 21:55:04.940304 23193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:55:04.941649 23118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:55:04.942360 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.942108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60528)
0620 21:55:04.942235 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.942292 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 21:55:04.943240 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.942941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60534)
0620 21:55:04.943100 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.943151 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260620 21:55:04.943434 23118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:44517: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
Jun 20 21:55:04 dist-test-slave-kk5c krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.253@KRBTEST.COM for kudu/127.0.193.253@KRBTEST.COM
I20260620 21:55:04.950222 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.940864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:41889 (local address 127.0.0.1:53448)
0620 21:55:04.941053 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.941069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:04.941148 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:04.941365 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:04.941368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:04.941540 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:04.941633 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:04.941640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.942586 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.942590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:04.943368 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:04.943379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.943985 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.943988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:04.944136 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:04.944736 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:04.944747 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:04.946325 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:04.948743 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:04.948747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:04.948749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:04.948944 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:04.949218 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:04.949221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:04.949223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:04.949302 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:04.949668 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:04.949671 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:04.949751 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 21:55:04.949866 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 21:55:04.950016 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 21:55:04.950235 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.940895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.0.193.253:41889)
0620 21:55:04.941149 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:04.941158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:04.941172 (+ 14us) server_negotiation.cc:407] Connection header received
0620 21:55:04.941217 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:04.941221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:04.941256 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:04.941356 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:04.941750 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.942443 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.943533 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:04.943800 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:04.943915 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:04.946553 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:04.946557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:04.946559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:04.946578 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:04.948612 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:04.949039 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:04.949045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:04.949047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:04.949125 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:04.949385 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:04.949389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:04.949391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:04.949576 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:04.949682 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:04.949917 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 21:55:04.950078 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":206,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.252.keytab.
Entry for principal kudu/127.0.193.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.252.keytab.
I20260620 21:55:04.951145 23193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:55:04.951956 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.951668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60544)
0620 21:55:04.951807 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.951895 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 21:55:04.951973 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.252:44517
--webserver_interface=127.0.193.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.252.keytab
--principal=kudu/127.0.193.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:36395,127.0.193.253:41889,127.0.193.252:44517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:04.952395 23144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
I20260620 21:55:04.953366 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.953107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60552)
0620 21:55:04.953210 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.953254 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 21:55:04.953532 23193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:44517: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
I20260620 21:55:04.974629 23118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } attempt: 1
I20260620 21:55:04.975565 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.975242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60556)
0620 21:55:04.975416 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.975482 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 21:55:04.976433 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:04.976193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60560)
0620 21:55:04.976309 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:04.976373 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 21:55:04.976606 23118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:44517: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
I20260620 21:55:05.002837 23193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } attempt: 1
I20260620 21:55:05.003742 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.003437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60576)
0620 21:55:05.003595 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:05.003668 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 21:55:05.004545 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.004347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60590)
0620 21:55:05.004450 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:05.004494 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 21:55:05.004743 23193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:44517: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
I20260620 21:55:05.051762 23118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } attempt: 2
I20260620 21:55:05.052613 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.052282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60594)
0620 21:55:05.052478 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:05.052538 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 21:55:05.053488 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.053260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60608)
0620 21:55:05.053379 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:05.053436 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 21:55:05.053670 23118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:44517: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
W20260620 21:55:05.065050 23201 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:55:05.065304 23201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:05.065357 23201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:05.065446 23201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:05.069550 23201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:05.069612 23201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:05.069631 23201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:05.069648 23201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:05.069703 23201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:05.072933 23193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } attempt: 2
I20260620 21:55:05.073896 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.073554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60624)
0620 21:55:05.073725 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:05.073789 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 21:55:05.074759 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.074518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60640)
0620 21:55:05.074654 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:05.074704 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 21:55:05.074918 23193 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:44517: Network error: Client connection negotiation failed: client connection to 127.0.193.252:44517: connect: Connection refused (error 111)
I20260620 21:55:05.075804 23201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:36395,127.0.193.253:41889,127.0.193.252:44517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/krb5kdc/kudu_127.0.193.252.keytab
--principal=kudu/127.0.193.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.252:44517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.193.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:05.076911 23201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:05.077945 23201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:05.086777 23207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:05.086820 23201 server_base.cc:1034] running on GCE node
W20260620 21:55:05.086777 23206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:05.086781 23209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:05.087858 23201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:05.089160 23201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:05.090436 23201 hybrid_clock.cc:648] HybridClock initialized: now 1781992505090414 us; error 39 us; skew 500 ppm
Jun 20 21:55:05 dist-test-slave-kk5c krb5kdc[23037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:05.093766 23201 init.cc:377] Logged in from keytab as kudu/127.0.193.252@KRBTEST.COM (short username kudu)
I20260620 21:55:05.094751 23201 webserver.cc:460] Webserver started at http://127.0.193.252:46405/ using document root <none> and password file <none>
I20260620 21:55:05.095384 23201 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:05.095458 23201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:05.095664 23201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:05.098032 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/data/instance:
uuid: "cb8410fbda3047598b73d8fb26d8ab60"
format_stamp: "Formatted at 2026-06-20 21:55:05 on dist-test-slave-kk5c"
server_key: "6fe4b80de7e1379dc3840c193c045f6a"
server_key_iv: "55d693405e5096f670d0137e5b218055"
server_key_version: "encryptionkey@0"
I20260620 21:55:05.098593 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/wal/instance:
uuid: "cb8410fbda3047598b73d8fb26d8ab60"
format_stamp: "Formatted at 2026-06-20 21:55:05 on dist-test-slave-kk5c"
server_key: "6fe4b80de7e1379dc3840c193c045f6a"
server_key_iv: "55d693405e5096f670d0137e5b218055"
server_key_version: "encryptionkey@0"
I20260620 21:55:05.102810 23201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:55:05.105530 23216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:05.106910 23201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:55:05.107059 23201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/wal
uuid: "cb8410fbda3047598b73d8fb26d8ab60"
format_stamp: "Formatted at 2026-06-20 21:55:05 on dist-test-slave-kk5c"
server_key: "6fe4b80de7e1379dc3840c193c045f6a"
server_key_iv: "55d693405e5096f670d0137e5b218055"
server_key_version: "encryptionkey@0"
I20260620 21:55:05.107241 23201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:05.118901 23201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:05.122974 23201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:05.123242 23201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:05.130939 23201 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.252:44517
I20260620 21:55:05.131402 23268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.252:44517 every 8 connection(s)
I20260620 21:55:05.132246 23201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/data/info.pb
I20260620 21:55:05.135277 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:05.139096 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 23201
I20260620 21:55:05.139277 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781992210835692-775-0/minicluster-data/master-2/wal/instance
I20260620 21:55:05.139567 775 external_mini_cluster.cc:1428] Setting key 45ce9227cdcb1db7e9ae2633162e7540
I20260620 21:55:05.139583 23269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:55:05 dist-test-slave-kk5c krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.252@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
Jun 20 21:55:05 dist-test-slave-kk5c krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.252@KRBTEST.COM
I20260620 21:55:05.152828 23118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } attempt: 3
I20260620 21:55:05.155166 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.141792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.0.193.254:36395)
0620 21:55:05.141941 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.141946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.142331 (+ 385us) server_negotiation.cc:407] Connection header received
0620 21:55:05.142827 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.142830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.142876 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.142948 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.144069 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.144617 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.146199 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.146565 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.146691 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.150424 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:05.150429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:05.150432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:05.150465 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:05.152659 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.153359 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.153363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.153365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.153467 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.153826 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.153828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.153830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.154100 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:05.154173 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:05.154817 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 21:55:05.154974 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:55:05.156463 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.141062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.0.193.252:44517)
0620 21:55:05.141515 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.141529 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.141562 (+ 33us) server_negotiation.cc:407] Connection header received
0620 21:55:05.142589 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.142613 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.142851 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.143044 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.143871 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.144835 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.146159 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.146866 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.146978 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.150446 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:05.150453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:05.150470 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:05.150493 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:05.152883 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.153549 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.153557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.153558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.153697 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.154009 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.154012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.154014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.154513 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:05.154741 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:05.154986 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 21:55:05.155618 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":288,"spinlock_wait_cycles":5504,"thread_start_us":139,"threads_started":1}
I20260620 21:55:05.156471 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.141726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36395 (local address 127.0.0.1:48038)
0620 21:55:05.142172 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:05.142229 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:05.142589 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:05.143025 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:05.143031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:05.143542 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:05.143824 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:05.143845 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.144752 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.144755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:05.146028 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:05.146040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.146803 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.146807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:05.146988 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.147797 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:05.147819 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:05.150225 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:05.152827 (+ 2602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:05.152835 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:05.152844 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:05.153194 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:05.153585 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:05.153588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:05.153590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:05.153711 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:05.154259 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:05.154267 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:05.154534 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 21:55:05.155358 (+ 824us) client_negotiation.cc:240] Negotiation successful
0620 21:55:05.155734 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":214,"thread_start_us":149,"threads_started":1}
I20260620 21:55:05.157856 23269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 21:55:05 dist-test-slave-kk5c krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for kudu/127.0.193.252@KRBTEST.COM
Jun 20 21:55:05 dist-test-slave-kk5c krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.252@KRBTEST.COM for kudu/127.0.193.253@KRBTEST.COM
I20260620 21:55:05.164656 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.153317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60650)
0620 21:55:05.153553 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:05.153570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:05.153651 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:05.153967 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:05.153970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:05.154143 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:05.154251 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:05.154258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.155652 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.155657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:05.156535 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:05.156549 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.157391 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.157396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:05.157481 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.158263 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:05.158274 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:05.160005 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:05.162962 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:05.162966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:05.162968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:05.163298 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:05.163586 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:05.163589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:05.163591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:05.163679 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:05.164079 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:05.164082 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:05.164188 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 21:55:05.164310 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 21:55:05.164468 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"spinlock_wait_cycles":1920}
I20260620 21:55:05.164705 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.153359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.0.193.252:44517)
0620 21:55:05.153727 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.153736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.153749 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:55:05.153785 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.153788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.153838 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.153931 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.154365 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.155146 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.156696 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.157066 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.157173 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.160188 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:05.160196 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:05.160202 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:05.160237 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:05.162834 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.163384 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.163387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.163389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.163485 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.163783 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.163787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.163789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.163977 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:05.164088 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:05.164379 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 21:55:05.164545 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":125,"threads_started":1}
I20260620 21:55:05.166116 23193 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } attempt: 3
Jun 20 21:55:05 dist-test-slave-kk5c krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:55:05.168448 23118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652: Bootstrap starting.
I20260620 21:55:05.169206 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.158559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.0.193.253:41889)
0620 21:55:05.158688 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.158693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.158896 (+ 203us) server_negotiation.cc:407] Connection header received
0620 21:55:05.159093 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.159097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.159147 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.159226 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.160312 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.161009 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.161833 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.162214 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.162336 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.164843 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:05.164849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:05.164854 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:05.164892 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:05.167373 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.167842 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.167846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.167848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.167935 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.168242 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.168245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.168247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.168468 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:05.168548 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:05.168899 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 21:55:05.169046 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":54}
I20260620 21:55:05.169690 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.158466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:41889 (local address 127.0.0.1:53456)
0620 21:55:05.158789 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:05.158808 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:05.158975 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:05.159408 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:05.159411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:05.159584 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:05.160176 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:05.160185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.161136 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.161140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:05.161698 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:05.161706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.162336 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.162340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:05.162451 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.162927 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:05.162943 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:05.164642 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:05.167472 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:05.167476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:05.167478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:05.167719 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:05.168050 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:05.168053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:05.168055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:05.168141 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:05.168625 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:05.168628 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:05.168718 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 21:55:05.169406 (+ 688us) client_negotiation.cc:240] Negotiation successful
0620 21:55:05.169548 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242}
I20260620 21:55:05.170450 23269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:55:05.171643 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.159259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.0.193.254:36395)
0620 21:55:05.159382 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.159386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.159444 (+ 58us) server_negotiation.cc:407] Connection header received
0620 21:55:05.159573 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.159576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.159617 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.159700 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.160745 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.161294 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.162730 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.163096 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.163235 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.167353 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:05.167357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:05.167360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:05.167385 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:05.169132 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.170222 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.170228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.170231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.170344 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.170693 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.170697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.170699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.170950 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:05.171139 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:05.171368 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 21:55:05.171494 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44,"spinlock_wait_cycles":4480}
Jun 20 21:55:05 dist-test-slave-kk5c krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.253@KRBTEST.COM for kudu/127.0.193.252@KRBTEST.COM
I20260620 21:55:05.172688 23118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:05.173768 23118 log.cc:824] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:05.176234 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.166544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60658)
0620 21:55:05.166730 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:05.166744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:05.166833 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:05.167114 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:05.167118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:05.167620 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:05.167740 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:05.167753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.168878 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.168882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:05.169743 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:05.169755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.170386 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.170390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:05.170502 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.170967 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:05.170980 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:05.172543 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:05.174553 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:05.174556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:05.174558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:05.174820 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:05.175148 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:05.175151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:05.175154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:05.175281 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:05.175731 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:05.175734 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:05.175825 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 21:55:05.175912 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 21:55:05.176063 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 21:55:05.176267 23118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652: No bootstrap required, opened a new log
I20260620 21:55:05.176841 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.166613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.0.193.252:44517)
0620 21:55:05.166723 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.166727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.166793 (+ 66us) server_negotiation.cc:407] Connection header received
0620 21:55:05.166925 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.166928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.166975 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.167229 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.167882 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.168743 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.169898 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.170257 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.170725 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.172696 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:05.172703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:05.172707 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:05.172734 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:05.174428 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.174953 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.174957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.174961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.175037 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.175416 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.175418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.175420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.175619 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:05.176506 (+ 887us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:05.176579 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 21:55:05.176689 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 21:55:05 dist-test-slave-kk5c krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.252@KRBTEST.COM for kudu/127.0.193.252@KRBTEST.COM
Jun 20 21:55:05 dist-test-slave-kk5c krb5kdc[23037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.253@KRBTEST.COM
I20260620 21:55:05.180459 23193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd: Bootstrap starting.
I20260620 21:55:05.181780 23118 raft_consensus.cc:348] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } }
I20260620 21:55:05.182087 23118 raft_consensus.cc:729] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67e8eed3e154e5ca95a6435386bf652, State: Initialized, Role: FOLLOWER
I20260620 21:55:05.182462 23118 consensus_queue.cc:260] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } }
I20260620 21:55:05.183720 23193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:05.184130 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.170977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:44517 (local address 127.0.0.1:60666)
0620 21:55:05.174330 (+ 3353us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:05.174358 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:05.174437 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:05.174820 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:05.174823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:05.174984 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:05.175080 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:05.175088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.176069 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.176073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:05.176797 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:05.176805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.177458 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.177461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:05.177541 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.177933 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:05.177946 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:05.180044 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:05.181958 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:05.181962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:05.181964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:05.182391 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:05.182802 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:05.182805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:05.182807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:05.182895 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:05.183485 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:05.183488 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:05.183599 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 21:55:05.183718 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 21:55:05.183932 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3261}
I20260620 21:55:05.184108 23118 sys_catalog.cc:564] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:05.184669 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.173379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.0.193.253:41889)
0620 21:55:05.173499 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.173503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.173516 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:55:05.173560 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.173564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.173607 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.173681 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.174548 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.175317 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.176412 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.176761 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.176836 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.180345 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:05.180349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:05.180352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:05.180380 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:05.182557 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.183130 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.183134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.183136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.183254 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.183593 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.183597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.183599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.183821 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:05.183979 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:05.184380 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 21:55:05.184523 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:55:05.184703 23193 log.cc:824] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:05.185256 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.171176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.0.193.252:44517)
0620 21:55:05.174567 (+ 3391us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.174570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.174581 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:55:05.174609 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.174615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.174653 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.174722 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.175206 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.175942 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.176930 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.177237 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.177342 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.180209 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:05.180213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:05.180215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:05.180239 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:05.181815 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.182497 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.182500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.182502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.182577 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.182992 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.182995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.182996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.183259 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:05.183440 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:05.184491 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0620 21:55:05.184609 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3331}
I20260620 21:55:05.186663 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } } }
I20260620 21:55:05.186846 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:05.187732 23193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd: No bootstrap required, opened a new log
I20260620 21:55:05.189746 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:55:05.191008 23092 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d67e8eed3e154e5ca95a6435386bf652"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48052
I20260620 21:55:05.191341 23092 raft_consensus.cc:482] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 21:55:05.191493 23092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:05.192807 23193 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } }
I20260620 21:55:05.193148 23193 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd49d46edb084ae68593d5d89c7666fd, State: Initialized, Role: FOLLOWER
I20260620 21:55:05.193154 23092 raft_consensus.cc:504] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } }
I20260620 21:55:05.193522 23193 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } }
I20260620 21:55:05.194141 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } } }
I20260620 21:55:05.194356 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:05.194638 23269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60: Bootstrap starting.
I20260620 21:55:05.195698 23193 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:05.195724 23167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d67e8eed3e154e5ca95a6435386bf652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fd49d46edb084ae68593d5d89c7666fd"
I20260620 21:55:05.196177 23167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:05.197770 23167 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67e8eed3e154e5ca95a6435386bf652 in term 1.
I20260620 21:55:05.198837 23269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:05.200335 23269 log.cc:824] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:05.201495 23067 leader_election.cc:302] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d67e8eed3e154e5ca95a6435386bf652, fd49d46edb084ae68593d5d89c7666fd; no voters:
I20260620 21:55:05.201839 23276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:05.202057 23276 raft_consensus.cc:686] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [term 1 LEADER]: Becoming Leader. State: Replica: d67e8eed3e154e5ca95a6435386bf652, State: Running, Role: LEADER
I20260620 21:55:05.202427 23276 consensus_queue.cc:237] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } }
W20260620 21:55:05.202965 23288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:55:05.203171 23288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 21:55:05.203758 23300 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:55:05.203931 23300 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:55:05.204324 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d67e8eed3e154e5ca95a6435386bf652. Latest consensus state: current_term: 1 leader_uuid: "d67e8eed3e154e5ca95a6435386bf652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } } }
I20260620 21:55:05.204486 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:05.207499 23306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:05.207826 23092 leader_election.cc:288] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [CANDIDATE]: Term 1 election: Requested vote from peers fd49d46edb084ae68593d5d89c7666fd (127.0.193.253:41889), cb8410fbda3047598b73d8fb26d8ab60 (127.0.193.252:44517)
I20260620 21:55:05.208647 23306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:05.194850 23050 master_runner.cc:186] Error getting master registration for 127.0.193.253:41889: OK, instance_id { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" instance_seqno: 1781992504913101 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:55:05.213359 23050 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:55:05.210961 23125 master_runner.cc:186] Error getting master registration for 127.0.193.252:44517: OK, instance_id { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" instance_seqno: 1781992505128717 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:55:05.215628 23125 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:55:05.221239 23167 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd [term 1 FOLLOWER]: Refusing update from remote peer d67e8eed3e154e5ca95a6435386bf652: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:05.222060 23276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:05.222718 23269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60: No bootstrap required, opened a new log
I20260620 21:55:05.226613 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d67e8eed3e154e5ca95a6435386bf652. Latest consensus state: current_term: 1 leader_uuid: "d67e8eed3e154e5ca95a6435386bf652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } } }
I20260620 21:55:05.226791 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:05.227042 23269 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } }
I20260620 21:55:05.227434 23269 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8410fbda3047598b73d8fb26d8ab60, State: Initialized, Role: FOLLOWER
I20260620 21:55:05.227949 23269 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } }
I20260620 21:55:05.230048 23243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d67e8eed3e154e5ca95a6435386bf652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb8410fbda3047598b73d8fb26d8ab60"
I20260620 21:55:05.230334 23269 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:05.230682 23242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:05.232960 23242 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60 [term 1 FOLLOWER]: Refusing update from remote peer d67e8eed3e154e5ca95a6435386bf652: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:05.233836 23304 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:05.234956 23243 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67e8eed3e154e5ca95a6435386bf652 in term 1.
I20260620 21:55:05.235924 23304 sys_catalog.cc:455] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d67e8eed3e154e5ca95a6435386bf652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } } }
I20260620 21:55:05.236053 23304 sys_catalog.cc:458] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:05.236595 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.215409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.0.193.252:44517)
0620 21:55:05.215552 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.215556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.223502 (+ 7946us) server_negotiation.cc:407] Connection header received
0620 21:55:05.223708 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.223712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.223765 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.223820 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.224856 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.225592 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.226778 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.227094 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.227200 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.230733 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:05.230738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:05.230740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:05.230771 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:05.232592 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.233131 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.233135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.233137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.233232 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.233589 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.233593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.233594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.233840 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:05.233947 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:05.236243 (+ 2296us) server_negotiation.cc:299] Negotiation successful
0620 21:55:05.236406 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:55:05.238162 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d67e8eed3e154e5ca95a6435386bf652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } } }
I20260620 21:55:05.238473 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd [sys.catalog]: This master's current role is: FOLLOWER
W20260620 21:55:05.244782 23330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:55:05.245221 23330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:55:05.245139 23317 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } } }
I20260620 21:55:05.245460 23317 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:05.248898 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.210722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.0.193.254:36395)
0620 21:55:05.210841 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.210847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.216367 (+ 5520us) server_negotiation.cc:407] Connection header received
0620 21:55:05.216701 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.216705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.216761 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.216817 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.218506 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.219332 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.239678 (+ 20346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.240136 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.240231 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.242483 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:05.242487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:05.242490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:05.242523 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:05.244421 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.247304 (+ 2883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.247308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.247310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.247428 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.247784 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.247787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.247789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.248051 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:05.248188 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:05.248580 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 21:55:05.248729 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 21:55:05.250576 23314 mvcc.cc:205] Tried to move back new op lower bound from 7299041301374042112 to 7299041301312917504. Current Snapshot: MvccSnapshot[applied={T|T < 7299041301374042112}]
I20260620 21:55:05.251088 23317 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d67e8eed3e154e5ca95a6435386bf652. Latest consensus state: current_term: 1 leader_uuid: "d67e8eed3e154e5ca95a6435386bf652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } } }
I20260620 21:55:05.251420 23317 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:05.252009 23315 mvcc.cc:205] Tried to move back new op lower bound from 7299041301374042112 to 7299041301312917504. Current Snapshot: MvccSnapshot[applied={T|T < 7299041301374042112}]
I20260620 21:55:05.253886 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:05.214869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.0.193.253:41889)
0620 21:55:05.215106 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.215112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.229448 (+ 14336us) server_negotiation.cc:407] Connection header received
0620 21:55:05.229649 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.229655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.229708 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.229767 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.234318 (+ 4551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.235028 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.237173 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:05.237504 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:05.237589 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:05.249842 (+ 12253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:05.249846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:05.249849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:05.249881 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:05.251807 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.252444 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.252448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.252450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.252541 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:05.252863 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:05.252866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:05.252868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:05.253095 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:05.253206 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:05.253587 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 21:55:05.253724 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 21:55:05.262266 23306 catalog_manager.cc:1347] Generated new cluster ID: 22d5a00ef503413681af81355ce39d7c
I20260620 21:55:05.262377 23306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:05.275848 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d67e8eed3e154e5ca95a6435386bf652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } } }
I20260620 21:55:05.276051 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652 [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:05.281164 23317 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d67e8eed3e154e5ca95a6435386bf652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67e8eed3e154e5ca95a6435386bf652" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36395 } } peers { permanent_uuid: "fd49d46edb084ae68593d5d89c7666fd" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 41889 } } peers { permanent_uuid: "cb8410fbda3047598b73d8fb26d8ab60" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 44517 } } }
I20260620 21:55:05.281481 23317 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:05.313503 23306 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:05.315167 23306 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:05.340415 23306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d67e8eed3e154e5ca95a6435386bf652: Generated new TSK 0
I20260620 21:55:05.341461 23306 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:55:05.377795 23272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:05.374925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.0.193.252:44517)
0620 21:55:05.375066 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.375073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.376516 (+ 1443us) server_negotiation.cc:407] Connection header received
0620 21:55:05.376759 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.376763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.376821 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.376881 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.377642 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42884: BlockingRecv error: recv got EOF from 127.0.0.1:42884 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 21:55:05.380666 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:05.375818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.0.193.253:41889)
0620 21:55:05.375946 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.375951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.378591 (+ 2640us) server_negotiation.cc:407] Connection header received
0620 21:55:05.379745 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.379749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.379810 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.379873 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.380516 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32856: BlockingRecv error: recv got EOF from 127.0.0.1:32856 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 21:55:05.382465 23120 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:05.378955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46966 (local address 127.0.193.254:36395)
0620 21:55:05.379089 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.379095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.381499 (+ 2404us) server_negotiation.cc:407] Connection header received
0620 21:55:05.381627 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.381631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.381694 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.381845 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.382313 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46966: BlockingRecv error: recv got EOF from 127.0.0.1:46966 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 21:55:05.388872 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:05.385984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32866 (local address 127.0.193.253:41889)
0620 21:55:05.386106 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.386111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.386673 (+ 562us) server_negotiation.cc:407] Connection header received
0620 21:55:05.386931 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.386935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.387013 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.387066 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.388719 (+ 1653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32866: BlockingRecv error: recv got EOF from 127.0.0.1:32866 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 21:55:05.397442 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:05.395136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.0.193.253:41889)
0620 21:55:05.395336 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.395341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.396189 (+ 848us) server_negotiation.cc:407] Connection header received
0620 21:55:05.396429 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.396433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.396490 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.396547 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.397301 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32878: BlockingRecv error: recv got EOF from 127.0.0.1:32878 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260620 21:55:05.404011 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:05.402261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.0.193.253:41889)
0620 21:55:05.402383 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.402401 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.402988 (+ 587us) server_negotiation.cc:407] Connection header received
0620 21:55:05.403217 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.403221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.403277 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.403332 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.403882 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32892: BlockingRecv error: recv got EOF from 127.0.0.1:32892 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 21:55:05.412968 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:05.410848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.0.193.253:41889)
0620 21:55:05.411019 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.411024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.411744 (+ 720us) server_negotiation.cc:407] Connection header received
0620 21:55:05.411972 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.411993 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.412063 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.412119 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.412830 (+ 711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32900: BlockingRecv error: recv got EOF from 127.0.0.1:32900 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260620 21:55:05.425872 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:05.423985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.0.193.253:41889)
0620 21:55:05.424132 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.424137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.424771 (+ 634us) server_negotiation.cc:407] Connection header received
0620 21:55:05.425009 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.425013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.425073 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.425132 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.425747 (+ 615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32914: BlockingRecv error: recv got EOF from 127.0.0.1:32914 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 21:55:05.446249 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:05.444907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.0.193.253:41889)
0620 21:55:05.445046 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.445052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.445248 (+ 196us) server_negotiation.cc:407] Connection header received
0620 21:55:05.445426 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.445429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.445491 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.445599 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.446111 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32924: BlockingRecv error: recv got EOF from 127.0.0.1:32924 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 21:55:05.482424 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:05.481053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.0.193.253:41889)
0620 21:55:05.481262 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.481268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.481585 (+ 317us) server_negotiation.cc:407] Connection header received
0620 21:55:05.481649 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.481651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.481702 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.481781 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.482287 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32938: BlockingRecv error: recv got EOF from 127.0.0.1:32938 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 21:55:05.550019 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:05.548783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.0.193.253:41889)
0620 21:55:05.548958 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.548963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.549092 (+ 129us) server_negotiation.cc:407] Connection header received
0620 21:55:05.549285 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.549293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.549340 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.549430 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.549904 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32946: BlockingRecv error: recv got EOF from 127.0.0.1:32946 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 21:55:05.681900 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:05.680486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.0.193.253:41889)
0620 21:55:05.680660 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.680664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.680881 (+ 217us) server_negotiation.cc:407] Connection header received
0620 21:55:05.680946 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.680949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.681097 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.681184 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.681781 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32956: BlockingRecv error: recv got EOF from 127.0.0.1:32956 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 21:55:05.941563 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:05.940474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.0.193.253:41889)
0620 21:55:05.940608 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:05.940612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:05.940721 (+ 109us) server_negotiation.cc:407] Connection header received
0620 21:55:05.940883 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:05.940886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:05.940934 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:05.941020 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:05.941425 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32962: BlockingRecv error: recv got EOF from 127.0.0.1:32962 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 21:55:06.205260 23300 catalog_manager.cc:1259] Loaded cluster ID: 22d5a00ef503413681af81355ce39d7c
I20260620 21:55:06.205382 23300 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd: loading cluster ID for follower catalog manager: success
I20260620 21:55:06.207315 23300 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd: acquiring CA information for follower catalog manager: success
I20260620 21:55:06.208600 23300 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd49d46edb084ae68593d5d89c7666fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:55:06.246670 23330 catalog_manager.cc:1259] Loaded cluster ID: 22d5a00ef503413681af81355ce39d7c
I20260620 21:55:06.246752 23330 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60: loading cluster ID for follower catalog manager: success
I20260620 21:55:06.248464 23330 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60: acquiring CA information for follower catalog manager: success
I20260620 21:55:06.249612 23330 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cb8410fbda3047598b73d8fb26d8ab60: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:55:06.460661 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:06.456219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.0.193.253:41889)
0620 21:55:06.456471 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:06.456476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:06.456494 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:55:06.456591 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:06.456594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:06.456727 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:06.456848 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:06.457125 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:06.457663 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:06.458857 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:06.459214 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:06.459377 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:06.459723 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:06.460181 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:06.460319 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:06.460382 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 21:55:06.460443 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":69,"threads_started":1}
I20260620 21:55:06.468451 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:06.464152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.0.193.252:44517)
0620 21:55:06.464401 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:06.464405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:06.464432 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:55:06.464516 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:06.464518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:06.464633 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:06.464739 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:06.465038 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:06.465595 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:06.466520 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:06.466843 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:06.467126 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:06.467517 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:06.467994 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:06.468158 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:06.468205 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 21:55:06.468270 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
I20260620 21:55:06.477147 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:06.472474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.0.193.253:41889)
0620 21:55:06.472766 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:06.472771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:06.473030 (+ 259us) server_negotiation.cc:407] Connection header received
0620 21:55:06.473301 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:06.473309 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:06.473391 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:06.473450 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:06.473905 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:06.474516 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:06.475975 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:06.476204 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:06.476318 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:06.476618 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:06.476712 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:06.476781 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:06.476909 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 21:55:06.476975 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260620 21:55:06.477870 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:06.472604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.0.193.252:44517)
0620 21:55:06.472751 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:06.472755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:06.473332 (+ 577us) server_negotiation.cc:407] Connection header received
0620 21:55:06.473524 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:06.473527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:06.473578 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:06.473661 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:06.474636 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:06.475312 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:06.476641 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:06.476989 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:06.477085 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:06.477413 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:06.477481 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:06.477566 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:06.477653 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 21:55:06.477710 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 21:55:06.478272 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:06.472637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.0.193.254:36395)
0620 21:55:06.472916 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:06.472921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:06.473222 (+ 301us) server_negotiation.cc:407] Connection header received
0620 21:55:06.473400 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:06.473403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:06.473583 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:06.473700 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:06.474142 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:06.474831 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:06.476131 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:06.476485 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:06.476613 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:06.477242 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:06.477637 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:06.477722 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:06.478050 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 21:55:06.478122 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260620 21:55:06.482003 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 23050
I20260620 21:55:06.488935 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 23125
I20260620 21:55:06.495477 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 23201
2026-06-20T21:55:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3217 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 21:55:06 dist-test-slave-kk5c krb5kdc[23415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 21:55:06 dist-test-slave-kk5c krb5kdc[23415](info): set up 2 sockets
Jun 20 21:55:06 dist-test-slave-kk5c krb5kdc[23415](info): commencing operation
krb5kdc: starting...
W20260620 21:55:07.587884 775 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/test-user.keytab.
Jun 20 21:55:07 dist-test-slave-kk5c krb5kdc[23415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T21:55:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T21:55:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
Entry for principal kudu/127.0.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab.
I20260620 21:55:07.723726 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36683
--webserver_interface=127.0.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:36683,127.0.193.253:40067,127.0.193.252:37415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 21:55:07.842694 23428 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:55:07.843070 23428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:07.843148 23428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:07.843298 23428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:07.847457 23428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:07.847573 23428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:07.847613 23428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:07.847651 23428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:07.847687 23428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:07.853186 23428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:36683,127.0.193.253:40067,127.0.193.252:37415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kudu_127.0.193.254.keytab
--principal=kudu/127.0.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.254:36683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:07.854388 23428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:07.855463 23428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:07.864316 23433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:07.864316 23434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:07.864578 23436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:07.864710 23428 server_base.cc:1034] running on GCE node
I20260620 21:55:07.865291 23428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:07.866545 23428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:07.867746 23428 hybrid_clock.cc:648] HybridClock initialized: now 1781992507867695 us; error 77 us; skew 500 ppm
Jun 20 21:55:07 dist-test-slave-kk5c krb5kdc[23415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:07.870788 23428 init.cc:377] Logged in from keytab as kudu/127.0.193.254@KRBTEST.COM (short username kudu)
I20260620 21:55:07.871836 23428 webserver.cc:460] Webserver started at http://127.0.193.254:40671/ using document root <none> and password file <none>
I20260620 21:55:07.872407 23428 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:07.872510 23428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:07.872746 23428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:07.874883 23428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/data/instance:
uuid: "17d9a6c6d9564f3bb586d0d70b9213cb"
format_stamp: "Formatted at 2026-06-20 21:55:07 on dist-test-slave-kk5c"
server_key: "758156e52f29597f0c114be331dfae1c"
server_key_iv: "72c883449098d073ea8dbb969b34d343"
server_key_version: "encryptionkey@0"
I20260620 21:55:07.875555 23428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/wal/instance:
uuid: "17d9a6c6d9564f3bb586d0d70b9213cb"
format_stamp: "Formatted at 2026-06-20 21:55:07 on dist-test-slave-kk5c"
server_key: "758156e52f29597f0c114be331dfae1c"
server_key_iv: "72c883449098d073ea8dbb969b34d343"
server_key_version: "encryptionkey@0"
I20260620 21:55:07.879873 23428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 21:55:07.882647 23443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:07.884008 23428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 21:55:07.884151 23428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/wal
uuid: "17d9a6c6d9564f3bb586d0d70b9213cb"
format_stamp: "Formatted at 2026-06-20 21:55:07 on dist-test-slave-kk5c"
server_key: "758156e52f29597f0c114be331dfae1c"
server_key_iv: "72c883449098d073ea8dbb969b34d343"
server_key_version: "encryptionkey@0"
I20260620 21:55:07.884280 23428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:07.898684 23428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:07.902048 23428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:07.902261 23428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:07.909801 23428 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.254:36683
I20260620 21:55:07.910140 23495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.254:36683 every 8 connection(s)
I20260620 21:55:07.910851 23428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/data/info.pb
I20260620 21:55:07.911059 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 23428
I20260620 21:55:07.911235 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-0/wal/instance
I20260620 21:55:07.911643 775 external_mini_cluster.cc:1428] Setting key 5fab7ccf05037355263b61c91bf58436
I20260620 21:55:07.914299 23496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:07.918284 23496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.193.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.253@KRBTEST.COM" created.
Jun 20 21:55:07 dist-test-slave-kk5c krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:55:07.937691 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:07.921817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.0.193.254:36683)
0620 21:55:07.922162 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:07.922172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:07.922212 (+ 40us) server_negotiation.cc:407] Connection header received
0620 21:55:07.923046 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:07.923054 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:07.923370 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:07.923555 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:07.925167 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:07.926677 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:07.928137 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:07.928864 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:07.929106 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:07.932150 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:07.932161 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:07.932183 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:07.932220 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:07.934164 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:07.934706 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:07.934736 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:07.934740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:07.934901 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:07.935354 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:07.935356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:07.935357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:07.935838 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:07.935953 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:07.936577 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 21:55:07.936837 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":231,"thread_start_us":148,"threads_started":1}
I20260620 21:55:07.937691 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:07.920699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36683 (local address 127.0.0.1:57580)
0620 21:55:07.921498 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:07.921544 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:07.922576 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:07.923624 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:07.923630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:07.924456 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:07.924889 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:07.924913 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:07.926875 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:07.926879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:07.927862 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:07.927874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:07.929140 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:07.929143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:07.929242 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:07.929958 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:07.929980 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:07.931941 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:07.934299 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:07.934311 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:07.934325 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:07.934586 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:07.935047 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:07.935050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:07.935051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:07.935179 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:07.935980 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:07.935987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:07.936245 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 21:55:07.936455 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 21:55:07.936914 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":100,"thread_start_us":282,"threads_started":1}
I20260620 21:55:07.940580 23496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:55:07.941452 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:07.941156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40067 (local address 127.0.0.1:41726)
0620 21:55:07.941269 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:07.941360 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 21:55:07.941646 23444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40067: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kudu_127.0.193.253.keytab.
Entry for principal kudu/127.0.193.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kudu_127.0.193.253.keytab.
I20260620 21:55:07.942497 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:07.942292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40067 (local address 127.0.0.1:41742)
0620 21:55:07.942400 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:07.942447 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 21:55:07.942647 23496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.253:40067: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40067: connect: Connection refused (error 111)
I20260620 21:55:07.942844 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.253:40067
--webserver_interface=127.0.193.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kudu_127.0.193.253.keytab
--principal=kudu/127.0.193.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:36683,127.0.193.253:40067,127.0.193.252:37415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:55:07.991868 23496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } attempt: 1
I20260620 21:55:07.992805 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:07.992506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40067 (local address 127.0.0.1:41752)
0620 21:55:07.992663 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:07.992725 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 21:55:07.993543 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:07.993331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40067 (local address 127.0.0.1:41754)
0620 21:55:07.993430 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:07.993473 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 21:55:07.993719 23496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.253:40067: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40067: connect: Connection refused (error 111)
W20260620 21:55:08.057121 23503 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:55:08.057463 23503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:08.057519 23503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:08.057588 23503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:08.061681 23503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:08.061770 23503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:08.061808 23503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:08.061869 23503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:08.061915 23503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:08.061879 23496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } attempt: 2
I20260620 21:55:08.062775 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.062472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40067 (local address 127.0.0.1:41766)
0620 21:55:08.062641 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.062701 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 21:55:08.063544 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.063320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40067 (local address 127.0.0.1:41778)
0620 21:55:08.063428 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.063481 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 21:55:08.063669 23496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.253:40067: Network error: Client connection negotiation failed: client connection to 127.0.193.253:40067: connect: Connection refused (error 111)
I20260620 21:55:08.067754 23503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:36683,127.0.193.253:40067,127.0.193.252:37415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kudu_127.0.193.253.keytab
--principal=kudu/127.0.193.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.253:40067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.193.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:08.068885 23503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:08.069960 23503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:08.078739 23509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:08.078816 23503 server_base.cc:1034] running on GCE node
W20260620 21:55:08.078739 23508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:08.078771 23511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:08.079663 23503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:08.080775 23503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:08.081964 23503 hybrid_clock.cc:648] HybridClock initialized: now 1781992508081944 us; error 32 us; skew 500 ppm
Jun 20 21:55:08 dist-test-slave-kk5c krb5kdc[23415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:08.085157 23503 init.cc:377] Logged in from keytab as kudu/127.0.193.253@KRBTEST.COM (short username kudu)
I20260620 21:55:08.086084 23503 webserver.cc:460] Webserver started at http://127.0.193.253:39587/ using document root <none> and password file <none>
I20260620 21:55:08.086655 23503 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:08.086719 23503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:08.086898 23503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:08.089419 23503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/data/instance:
uuid: "821726f71c91403981d2eee34a1f6c96"
format_stamp: "Formatted at 2026-06-20 21:55:08 on dist-test-slave-kk5c"
server_key: "5e404de14eb81477d100ce5abed22bce"
server_key_iv: "b306d40798d3ac3f4e0215d5da5b25b1"
server_key_version: "encryptionkey@0"
I20260620 21:55:08.089984 23503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/wal/instance:
uuid: "821726f71c91403981d2eee34a1f6c96"
format_stamp: "Formatted at 2026-06-20 21:55:08 on dist-test-slave-kk5c"
server_key: "5e404de14eb81477d100ce5abed22bce"
server_key_iv: "b306d40798d3ac3f4e0215d5da5b25b1"
server_key_version: "encryptionkey@0"
I20260620 21:55:08.094318 23503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:55:08.097249 23518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:08.098825 23503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 21:55:08.099005 23503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/wal
uuid: "821726f71c91403981d2eee34a1f6c96"
format_stamp: "Formatted at 2026-06-20 21:55:08 on dist-test-slave-kk5c"
server_key: "5e404de14eb81477d100ce5abed22bce"
server_key_iv: "b306d40798d3ac3f4e0215d5da5b25b1"
server_key_version: "encryptionkey@0"
I20260620 21:55:08.099139 23503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:08.110927 23503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:08.114679 23503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:08.114923 23503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:08.123289 23503 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.253:40067
I20260620 21:55:08.123625 23570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.253:40067 every 8 connection(s)
I20260620 21:55:08.124584 23503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/data/info.pb
I20260620 21:55:08.128517 23571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 21:55:08.129837 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 23503
I20260620 21:55:08.130025 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-1/wal/instance
I20260620 21:55:08.130318 775 external_mini_cluster.cc:1428] Setting key 746a67cb64923e5dfb2ae47094f801e4
I20260620 21:55:08.133730 23571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.193.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.193.252@KRBTEST.COM" created.
Jun 20 21:55:08 dist-test-slave-kk5c krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.253@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:55:08.152748 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.136616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36683 (local address 127.0.0.1:57590)
0620 21:55:08.137228 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.137274 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:08.138265 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:08.138869 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:08.138877 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:08.139356 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:08.139898 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.139977 (+ 79us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.140914 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.140918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.141873 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.141884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.142544 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.142548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.142789 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.143645 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:08.143672 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:08.145896 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:08.148667 (+ 2771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.148674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.148690 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.149021 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.149502 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.149505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.149506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.149679 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.150315 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:08.150323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:08.151309 (+ 986us) client_negotiation.cc:769] Sending connection context
0620 21:55:08.151551 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 21:55:08.151864 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":188,"threads_started":1}
I20260620 21:55:08.152962 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.136059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.0.193.254:36683)
0620 21:55:08.136217 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.136220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.137472 (+ 1252us) server_negotiation.cc:407] Connection header received
0620 21:55:08.138500 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.138504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.138576 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.138679 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.140203 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.140760 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.142095 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.142358 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.142493 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.146126 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:08.146130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:08.146132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:08.146160 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:08.148282 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.149165 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.149173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.149176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.149306 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.149822 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.149828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.149832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.150150 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:08.150292 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:08.152500 (+ 2208us) server_negotiation.cc:299] Negotiation successful
0620 21:55:08.152684 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 21:55:08.154531 23571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:55:08.154874 23496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } attempt: 3
Jun 20 21:55:08 dist-test-slave-kk5c krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for kudu/127.0.193.253@KRBTEST.COM
Jun 20 21:55:08 dist-test-slave-kk5c krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.253@KRBTEST.COM for kudu/127.0.193.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.193.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kudu_127.0.193.252.keytab.
Entry for principal kudu/127.0.193.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kudu_127.0.193.252.keytab.
I20260620 21:55:08.168062 775 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
/tmp/dist-test-taskBo3tyv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.252:37415
--webserver_interface=127.0.193.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kudu_127.0.193.252.keytab
--principal=kudu/127.0.193.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.193.212:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.193.254:36683,127.0.193.253:40067,127.0.193.252:37415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 21:55:08.169582 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.155379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40067 (local address 127.0.0.1:41784)
0620 21:55:08.155585 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.155600 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:08.155725 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:08.156602 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:08.156605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:08.156814 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:08.156957 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.156965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.158380 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.158383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.158915 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.158923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.159938 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.159944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.160052 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.160735 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:08.160775 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:08.163983 (+ 3208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:08.167524 (+ 3541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.167529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.167533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.167843 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.168296 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.168299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.168301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.168382 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.168999 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:08.169002 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:08.169110 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 21:55:08.169224 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 21:55:08.169375 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":3}
I20260620 21:55:08.169626 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.155301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.0.193.253:40067)
0620 21:55:08.155941 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.156028 (+ 87us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.156060 (+ 32us) server_negotiation.cc:407] Connection header received
0620 21:55:08.156114 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.156121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.156371 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.156533 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.157111 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.158251 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.159062 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.159759 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.159899 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.164181 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:08.164185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:08.164186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:08.164207 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:08.167304 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.168077 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.168084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.168087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.168196 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.168512 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.168515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.168517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.168902 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:08.169045 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:08.169321 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 21:55:08.169459 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":289,"server-negotiator.queue_time_us":417,"spinlock_wait_cycles":10112,"thread_start_us":190,"threads_started":1}
I20260620 21:55:08.169963 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.155346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40067 (local address 127.0.0.1:41798)
0620 21:55:08.155559 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.155576 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:08.155667 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:08.156602 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:08.156605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:08.156814 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:08.156957 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.156970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.158237 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.158241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.159181 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.159376 (+ 195us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.159981 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.159984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.160114 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.160828 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:08.160846 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:08.163167 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:08.167657 (+ 4490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.167660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.167663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.168063 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.168693 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.168696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.168698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.168813 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.169350 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:08.169356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:08.169501 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 21:55:08.169609 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 21:55:08.169798 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 21:55:08.170559 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.155407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.0.193.253:40067)
0620 21:55:08.156193 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.156203 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.156214 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:55:08.156242 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.156244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.156371 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.156533 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.157136 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.158066 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.159518 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.159856 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.159981 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.163418 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:08.163430 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:08.163445 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:08.163474 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:08.167535 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.168458 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.168464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.168468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.168577 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.168930 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.168936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.168940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.169222 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:08.170262 (+ 1040us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:08.170324 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 21:55:08.170435 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":722,"spinlock_wait_cycles":8192,"thread_start_us":121,"threads_started":1}
I20260620 21:55:08.171921 23496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:55:08.172087 23571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:55:08.172780 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.172478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41576)
0620 21:55:08.172642 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.172691 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 21:55:08.172947 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.172458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41574)
0620 21:55:08.172763 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.172858 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
W20260620 21:55:08.173213 23520 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
I20260620 21:55:08.174209 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.173975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41588)
0620 21:55:08.174114 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.174161 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 21:55:08.174535 23571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:37415: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
I20260620 21:55:08.174858 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.174538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41596)
0620 21:55:08.174750 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.174795 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260620 21:55:08.175017 23496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:37415: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
I20260620 21:55:08.206202 23496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } attempt: 1
I20260620 21:55:08.207346 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.206830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41606)
0620 21:55:08.207047 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.207128 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 21:55:08.208251 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.208002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41612)
0620 21:55:08.208132 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.208192 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 21:55:08.208392 23496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:37415: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
I20260620 21:55:08.223840 23571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } attempt: 1
I20260620 21:55:08.224984 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.224654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41624)
0620 21:55:08.224802 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.224870 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 21:55:08.225783 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.225543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41634)
0620 21:55:08.225658 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.225711 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 21:55:08.225973 23571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:37415: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
I20260620 21:55:08.283618 23496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } attempt: 2
I20260620 21:55:08.284787 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.284241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41650)
0620 21:55:08.284502 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.284578 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260620 21:55:08.285797 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.285486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41664)
0620 21:55:08.285631 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.285705 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 21:55:08.286023 23496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:37415: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
I20260620 21:55:08.294164 23571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } attempt: 2
I20260620 21:55:08.295236 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.294794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41670)
0620 21:55:08.294981 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.295075 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 21:55:08.296149 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.295836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41686)
0620 21:55:08.295985 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.296057 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 21:55:08.296336 23571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:37415: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
W20260620 21:55:08.306602 23578 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 21:55:08.307003 23578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 21:55:08.307081 23578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 21:55:08.307170 23578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 21:55:08.311950 23578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 21:55:08.312057 23578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 21:55:08.312098 23578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 21:55:08.312141 23578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 21:55:08.312181 23578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 21:55:08.318495 23578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.193.212:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.193.254:36683,127.0.193.253:40067,127.0.193.252:37415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/krb5kdc/kudu_127.0.193.252.keytab
--principal=kudu/127.0.193.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.193.252:37415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.193.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 20:53:00 UTC on ccycloud.quasar-dvwnvj.root.comops.site
build id 4457723
I20260620 21:55:08.319844 23578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 21:55:08.320971 23578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 21:55:08.330480 23585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 21:55:08.330574 23584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:08.330816 23578 server_base.cc:1034] running on GCE node
W20260620 21:55:08.330479 23587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 21:55:08.331820 23578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 21:55:08.333179 23578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 21:55:08.334436 23578 hybrid_clock.cc:648] HybridClock initialized: now 1781992508334362 us; error 36 us; skew 500 ppm
Jun 20 21:55:08 dist-test-slave-kk5c krb5kdc[23415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 21:55:08.337940 23578 init.cc:377] Logged in from keytab as kudu/127.0.193.252@KRBTEST.COM (short username kudu)
I20260620 21:55:08.339025 23578 webserver.cc:460] Webserver started at http://127.0.193.252:42655/ using document root <none> and password file <none>
I20260620 21:55:08.339730 23578 fs_manager.cc:359] Metadata directory not provided
I20260620 21:55:08.339799 23578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 21:55:08.340034 23578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 21:55:08.342217 23578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/data/instance:
uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3"
format_stamp: "Formatted at 2026-06-20 21:55:08 on dist-test-slave-kk5c"
server_key: "d822e6ed81400a073ac9b309e06e6cd2"
server_key_iv: "ca727864d347a6b9b4d3180f0004f2fa"
server_key_version: "encryptionkey@0"
I20260620 21:55:08.342818 23578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/wal/instance:
uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3"
format_stamp: "Formatted at 2026-06-20 21:55:08 on dist-test-slave-kk5c"
server_key: "d822e6ed81400a073ac9b309e06e6cd2"
server_key_iv: "ca727864d347a6b9b4d3180f0004f2fa"
server_key_version: "encryptionkey@0"
I20260620 21:55:08.346793 23578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 21:55:08.349702 23594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 21:55:08.351431 23578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 21:55:08.351608 23578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/data,/tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/wal
uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3"
format_stamp: "Formatted at 2026-06-20 21:55:08 on dist-test-slave-kk5c"
server_key: "d822e6ed81400a073ac9b309e06e6cd2"
server_key_iv: "ca727864d347a6b9b4d3180f0004f2fa"
server_key_version: "encryptionkey@0"
I20260620 21:55:08.351739 23578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 21:55:08.375243 23578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 21:55:08.378548 23578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 21:55:08.378767 23578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 21:55:08.385214 23496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } attempt: 3
I20260620 21:55:08.386068 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.385779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41690)
0620 21:55:08.385919 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.385993 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 21:55:08.386857 23578 rpc_server.cc:307] RPC server started. Bound to: 127.0.193.252:37415
I20260620 21:55:08.386904 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.386599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41702)
0620 21:55:08.386760 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.386823 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260620 21:55:08.387063 23496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.193.252:37415: Network error: Client connection negotiation failed: client connection to 127.0.193.252:37415: connect: Connection refused (error 111)
I20260620 21:55:08.387229 23646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.193.252:37415 every 8 connection(s)
I20260620 21:55:08.387480 23571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } attempt: 3
I20260620 21:55:08.388221 23578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/data/info.pb
I20260620 21:55:08.391528 23647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 21:55:08 dist-test-slave-kk5c krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.253@KRBTEST.COM for kudu/127.0.193.252@KRBTEST.COM
I20260620 21:55:08.396474 23647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:55:08.396855 775 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu as pid 23578
I20260620 21:55:08.396991 775 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBo3tyv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781992210835692-775-0/minicluster-data/master-2/wal/instance
I20260620 21:55:08.397344 775 external_mini_cluster.cc:1428] Setting key f208ccc7ab6a202d10e39923ca4446f8
I20260620 21:55:08.402696 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.387947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41706)
0620 21:55:08.388158 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.388176 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:08.388281 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:08.390887 (+ 2606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:08.390891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:08.391134 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:08.391299 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.391308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.392811 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.392818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.393635 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.393650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.394495 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.394500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.394605 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.395165 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:08.395181 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:08.397031 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:08.400299 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.400303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.400306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.400592 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.401021 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.401025 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.401028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.401136 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.401982 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:08.401986 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:08.402107 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 21:55:08.402259 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 21:55:08.402434 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 21:55:08.404773 23649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.388898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.0.193.252:37415)
0620 21:55:08.389303 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.389323 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.389373 (+ 50us) server_negotiation.cc:407] Connection header received
0620 21:55:08.390175 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.390197 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.390566 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.390920 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.391568 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.392644 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.393784 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.394276 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.394483 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.397277 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:08.397283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:08.397301 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:08.397335 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:08.400159 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.400693 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.400699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.400706 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.400885 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.401257 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.401261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.401263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.401858 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:08.402014 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:08.403073 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0620 21:55:08.403476 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":144,"threads_started":1}
Jun 20 21:55:08 dist-test-slave-kk5c krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.252@KRBTEST.COM
Jun 20 21:55:08 dist-test-slave-kk5c krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.252@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
I20260620 21:55:08.410964 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.398694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.0.193.254:36683)
0620 21:55:08.398867 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.398873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.399325 (+ 452us) server_negotiation.cc:407] Connection header received
0620 21:55:08.399577 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.399583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.399646 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.399748 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.400622 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.401446 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.402723 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.403110 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.403255 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.406287 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:08.406292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:08.406295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:08.406331 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:08.408539 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.409210 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.409214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.409216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.409331 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.409718 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.409725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.409729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.410043 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:08.410156 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:08.410618 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 21:55:08.410782 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 21:55:08.411031 23571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96: Bootstrap starting.
I20260620 21:55:08.411291 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.398134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.0.193.252:37415)
0620 21:55:08.398864 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.398874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.398887 (+ 13us) server_negotiation.cc:407] Connection header received
0620 21:55:08.398922 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.398925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.398970 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.399049 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.400121 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.400884 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.401813 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.402161 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.402259 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.406240 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:08.406244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:08.406246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:08.406268 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:08.408947 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.409476 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.409483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.409486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.409642 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.409978 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.409981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.409983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.410223 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:08.410428 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:08.410686 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 21:55:08.410825 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"spinlock_wait_cycles":1792,"thread_start_us":137,"threads_started":1}
I20260620 21:55:08.411641 23652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.398593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.254:36683 (local address 127.0.0.1:57602)
0620 21:55:08.399140 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.399238 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:08.399440 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:08.399807 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:08.399816 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:08.400357 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:08.400487 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.400494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.401586 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.401589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.402513 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.402520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.403265 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.403269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.403404 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.404009 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:08.404020 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:08.406032 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:08.408703 (+ 2671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.408713 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.408718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.409085 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.409477 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.409481 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.409483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.409601 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.410176 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:08.410182 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:08.410325 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 21:55:08.411235 (+ 910us) client_negotiation.cc:240] Negotiation successful
0620 21:55:08.411458 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":84,"threads_started":1}
I20260620 21:55:08.413507 23647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } has no permanent_uuid. Determining permanent_uuid...
I20260620 21:55:08.414500 23571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:08.415505 23571 log.cc:824] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:08.417896 23571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96: No bootstrap required, opened a new log
Jun 20 21:55:08 dist-test-slave-kk5c krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.254@KRBTEST.COM
Jun 20 21:55:08 dist-test-slave-kk5c krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.252@KRBTEST.COM for kudu/127.0.193.253@KRBTEST.COM
I20260620 21:55:08.421908 23571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } }
I20260620 21:55:08.422209 23571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 821726f71c91403981d2eee34a1f6c96, State: Initialized, Role: FOLLOWER
I20260620 21:55:08.422578 23571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } }
I20260620 21:55:08.423161 23654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } } }
I20260620 21:55:08.423482 23654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:08.424464 23652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.414105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.253:40067 (local address 127.0.0.1:41810)
0620 21:55:08.414273 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.414289 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:08.414370 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:08.414698 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:08.414701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:08.414887 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:08.415021 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.415029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.416225 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.416228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.417026 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.417038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.417678 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.417684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.417830 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.418469 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:08.418485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:08.420341 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:08.422481 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.422497 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.422499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.422764 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.423095 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.423098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.423099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.423219 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.423698 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:08.423702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:08.423855 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 21:55:08.423993 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 21:55:08.424183 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 21:55:08.424485 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.414205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.0.193.253:40067)
0620 21:55:08.414351 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.414357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.414372 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:55:08.414460 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.414463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.414512 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.414598 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.415164 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.416091 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.417180 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.417537 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.417647 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.420537 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:08.420541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:08.420546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:08.420575 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:08.422349 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.422887 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.422893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.422898 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.422985 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.423323 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.423326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.423328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.423597 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:08.423703 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:08.424073 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 21:55:08.424198 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 21:55:08.424580 23571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:08.425984 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.412872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.0.193.254:36683)
0620 21:55:08.412993 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.412998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.413060 (+ 62us) server_negotiation.cc:407] Connection header received
0620 21:55:08.413226 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.413229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.413327 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.413405 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.414363 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.415054 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.416383 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.416639 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.416767 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.420602 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:08.420606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:08.420609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:08.420646 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:08.422763 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.423391 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.423396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.423398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.423495 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.423861 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.423864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.423866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.424324 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:08.424655 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:08.425589 (+ 934us) server_negotiation.cc:299] Negotiation successful
0620 21:55:08.425748 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 21:55:08.426541 23647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } has no permanent_uuid. Determining permanent_uuid...
W20260620 21:55:08.431455 23665 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:55:08.431551 23665 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 20 21:55:08 dist-test-slave-kk5c krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.252@KRBTEST.COM for kudu/127.0.193.252@KRBTEST.COM
Jun 20 21:55:08 dist-test-slave-kk5c krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.193.253@KRBTEST.COM
I20260620 21:55:08.429720 23503 master_runner.cc:186] Error getting master registration for 127.0.193.254:36683: OK, instance_id { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" instance_seqno: 1781992507907692 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:55:08.434031 23503 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:55:08.438164 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.427172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.0.193.253:40067)
0620 21:55:08.427299 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.427303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.427318 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:55:08.427461 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.427465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.427516 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.427599 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.428616 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.429397 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.430639 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.431003 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.431085 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.434361 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:08.434365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:08.434368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:08.434399 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:08.436124 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.436722 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.436729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.436731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.436836 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.437154 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.437158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.437160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.437405 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:08.437601 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:08.437861 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 21:55:08.437991 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 21:55:08.438164 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.427346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41722 (local address 127.0.193.252:37415)
0620 21:55:08.427456 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.427461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.427472 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:55:08.427530 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.427533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.427582 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.427648 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.428130 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.428879 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.429887 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.430237 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.430424 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.433576 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:08.433583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:08.433586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:08.433619 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:08.436224 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.436827 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.436830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.436832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.436938 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.437232 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.437235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.437237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.437508 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:08.437596 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:08.437918 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 21:55:08.438031 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 21:55:08.438604 23652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.427119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41722)
0620 21:55:08.427360 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.427373 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:08.427448 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:08.427764 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:08.427766 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:08.427904 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:08.428009 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.428016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.429004 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.429007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.429754 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.429765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.430575 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.430578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.430654 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.431223 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:08.431238 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:08.433356 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:08.436444 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.436447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.436449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.436703 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.437028 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.437031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.437033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.437139 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.437635 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:08.437638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:08.437725 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 21:55:08.438318 (+ 593us) client_negotiation.cc:240] Negotiation successful
0620 21:55:08.438459 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260620 21:55:08.441613 23647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3: Bootstrap starting.
I20260620 21:55:08.444312 23647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:08.445209 23647 log.cc:824] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:08.446770 775 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 21:55:08.447615 23647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3: No bootstrap required, opened a new log
I20260620 21:55:08.448547 23470 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57614
I20260620 21:55:08.451092 23647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } }
I20260620 21:55:08.451481 23647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a091053f9044a1fa0f6c22e7ddcf5d3, State: Initialized, Role: FOLLOWER
I20260620 21:55:08.451840 23647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } }
I20260620 21:55:08.452512 23668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } } }
I20260620 21:55:08.452766 23668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:08.453570 23647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 21:55:08.460161 23679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:55:08.460372 23679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:55:08.458503 23578 master_runner.cc:186] Error getting master registration for 127.0.193.254:36683: OK, instance_id { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" instance_seqno: 1781992507907692 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 21:55:08.461934 23578 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:55:08.551213 23496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } attempt: 4
Jun 20 21:55:08 dist-test-slave-kk5c krb5kdc[23415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781992507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.193.254@KRBTEST.COM for kudu/127.0.193.252@KRBTEST.COM
I20260620 21:55:08.561821 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.551905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.0.193.252:37415)
0620 21:55:08.552048 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.552054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.552120 (+ 66us) server_negotiation.cc:407] Connection header received
0620 21:55:08.552293 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.552297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.552349 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.552448 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.552998 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.553587 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.554731 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.555277 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.555418 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.558021 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:08.558025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:08.558030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:08.558062 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:08.560011 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.560481 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.560485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.560487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.560579 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.560873 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.560876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.560878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.561125 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:08.561243 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:08.561525 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 21:55:08.561640 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 21:55:08.561821 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.551843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.193.252:37415 (local address 127.0.0.1:41724)
0620 21:55:08.552047 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 21:55:08.552062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 21:55:08.552171 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 21:55:08.552474 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 21:55:08.552476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 21:55:08.552745 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 21:55:08.552866 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.552872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.553707 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.553712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.554578 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 21:55:08.554607 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.555591 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.555594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 21:55:08.555659 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.556247 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 21:55:08.556265 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 21:55:08.557861 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 21:55:08.560134 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.560137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.560139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.560361 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.560675 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 21:55:08.560681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 21:55:08.560684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 21:55:08.560785 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 21:55:08.561246 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 21:55:08.561251 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 21:55:08.561360 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 21:55:08.561466 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 21:55:08.561641 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260620 21:55:08.565791 23496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb: Bootstrap starting.
I20260620 21:55:08.568972 23496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb: Neither blocks nor log segments found. Creating new log.
I20260620 21:55:08.569902 23496 log.cc:824] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb: Log is configured to *not* fsync() on all Append() calls
I20260620 21:55:08.572271 23496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb: No bootstrap required, opened a new log
I20260620 21:55:08.575608 23496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } }
I20260620 21:55:08.575860 23496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d9a6c6d9564f3bb586d0d70b9213cb, State: Initialized, Role: FOLLOWER
I20260620 21:55:08.576210 23496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } }
I20260620 21:55:08.576809 23682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } } }
I20260620 21:55:08.577165 23682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:08.577737 23496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 21:55:08.577775 23470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 21:55:08.577883 23470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:08.579490 23470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } }
I20260620 21:55:08.580611 23470 leader_election.cc:288] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [CANDIDATE]: Term 1 election: Requested vote from peers 821726f71c91403981d2eee34a1f6c96 (127.0.193.253:40067), 7a091053f9044a1fa0f6c22e7ddcf5d3 (127.0.193.252:37415)
I20260620 21:55:08.582124 23539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "821726f71c91403981d2eee34a1f6c96"
I20260620 21:55:08.582410 23539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:08.584167 23539 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d9a6c6d9564f3bb586d0d70b9213cb in term 1.
W20260620 21:55:08.584395 23693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 21:55:08.584475 23693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 21:55:08.584734 23621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3"
I20260620 21:55:08.584978 23621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 21:55:08.586270 23621 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d9a6c6d9564f3bb586d0d70b9213cb in term 1.
I20260620 21:55:08.586952 23445 leader_election.cc:302] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d9a6c6d9564f3bb586d0d70b9213cb, 7a091053f9044a1fa0f6c22e7ddcf5d3; no voters:
I20260620 21:55:08.590188 23682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 21:55:08.590894 23682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [term 1 LEADER]: Becoming Leader. State: Replica: 17d9a6c6d9564f3bb586d0d70b9213cb, State: Running, Role: LEADER
I20260620 21:55:08.591390 23682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } }
I20260620 21:55:08.593711 23682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17d9a6c6d9564f3bb586d0d70b9213cb. Latest consensus state: current_term: 1 leader_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } } }
I20260620 21:55:08.594799 23682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:08.597311 23428 master_runner.cc:263] No leader master found from master 17d9a6c6d9564f3bb586d0d70b9213cb
I20260620 21:55:08.597410 23428 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 21:55:08.598186 23704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 21:55:08.599403 23704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 21:55:08.608891 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.588948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57616 (local address 127.0.193.254:36683)
0620 21:55:08.589507 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.589511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.597767 (+ 8256us) server_negotiation.cc:407] Connection header received
0620 21:55:08.597940 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.597944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.597999 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.598073 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.599135 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.599903 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.601118 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.601461 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.601544 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.603783 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:08.603787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:08.603790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:08.603821 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:08.606097 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.607131 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.607136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.607138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.607272 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.607653 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.607658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.607660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.607986 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:08.608109 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:08.608527 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 21:55:08.608683 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464}
W20260620 21:55:08.617128 23524 tablet.cc:2369] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:55:08.620743 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.589137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.0.193.253:40067)
0620 21:55:08.589257 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.589261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.605279 (+ 16018us) server_negotiation.cc:407] Connection header received
0620 21:55:08.611401 (+ 6122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.611408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.611469 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.611535 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.612294 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.613111 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.614360 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.614702 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.614785 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.616848 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:08.616854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:08.616857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:08.616889 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:08.618675 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.619215 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.619219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.619222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.619325 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.619688 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.619693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.619694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.619928 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:08.620049 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:08.620439 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 21:55:08.620582 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:55:08.626245 23621 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3 [term 1 FOLLOWER]: Refusing update from remote peer 17d9a6c6d9564f3bb586d0d70b9213cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 21:55:08.627143 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:08.594282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.0.193.252:37415)
0620 21:55:08.594423 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.594429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.605597 (+ 11168us) server_negotiation.cc:407] Connection header received
0620 21:55:08.609548 (+ 3951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.609554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.609625 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.609695 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.612374 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.613125 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.618143 (+ 5018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:08.618544 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:08.618643 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:08.622249 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 21:55:08.622254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 21:55:08.622258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 21:55:08.622295 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 21:55:08.624415 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.625191 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.625198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.625203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.625316 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 21:55:08.625720 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 21:55:08.625724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 21:55:08.625726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 21:55:08.625970 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 21:55:08.626148 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:08.626719 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 21:55:08.626868 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 21:55:08.628902 23539 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96 [term 1 FOLLOWER]: Refusing update from remote peer 17d9a6c6d9564f3bb586d0d70b9213cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 21:55:08.630391 23600 tablet.cc:2369] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 21:55:08.630465 23701 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:08.635528 23668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17d9a6c6d9564f3bb586d0d70b9213cb. Latest consensus state: current_term: 1 leader_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } } }
I20260620 21:55:08.635694 23668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:08.636731 23701 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 21:55:08.641781 23654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17d9a6c6d9564f3bb586d0d70b9213cb. Latest consensus state: current_term: 1 leader_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } } }
I20260620 21:55:08.641957 23654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:08.646687 23701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } } }
I20260620 21:55:08.646854 23701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:08.647135 23705 mvcc.cc:205] Tried to move back new op lower bound from 7299041315323604992 to 7299041315195215872. Current Snapshot: MvccSnapshot[applied={T|T < 7299041315323604992}]
I20260620 21:55:08.649587 23668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } } }
I20260620 21:55:08.649752 23668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:08.649993 23701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } } }
I20260620 21:55:08.650116 23701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb [sys.catalog]: This master's current role is: LEADER
I20260620 21:55:08.651643 23654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17d9a6c6d9564f3bb586d0d70b9213cb" member_type: VOTER last_known_addr { host: "127.0.193.254" port: 36683 } } peers { permanent_uuid: "821726f71c91403981d2eee34a1f6c96" member_type: VOTER last_known_addr { host: "127.0.193.253" port: 40067 } } peers { permanent_uuid: "7a091053f9044a1fa0f6c22e7ddcf5d3" member_type: VOTER last_known_addr { host: "127.0.193.252" port: 37415 } } }
I20260620 21:55:08.651787 23654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 21:55:08.652616 23708 mvcc.cc:205] Tried to move back new op lower bound from 7299041315323604992 to 7299041315195215872. Current Snapshot: MvccSnapshot[applied={T|T < 7299041315323604992}]
I20260620 21:55:08.655115 23704 catalog_manager.cc:1347] Generated new cluster ID: bc3d2fe147ee42cabace4f7c5e9e55b1
I20260620 21:55:08.655217 23704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 21:55:08.677249 23704 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 21:55:08.678736 23704 catalog_manager.cc:1504] Loading token signing keys...
I20260620 21:55:08.689635 23704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17d9a6c6d9564f3bb586d0d70b9213cb: Generated new TSK 0
I20260620 21:55:08.690562 23704 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 21:55:08.717334 23500 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:08.715710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57624 (local address 127.0.193.254:36683)
0620 21:55:08.715865 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.715870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.716110 (+ 240us) server_negotiation.cc:407] Connection header received
0620 21:55:08.716356 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.716359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.716418 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.716476 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.717159 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57624: BlockingRecv error: recv got EOF from 127.0.0.1:57624 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 21:55:08.717387 23651 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:08.715796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.0.193.252:37415)
0620 21:55:08.715934 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.715939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.716342 (+ 403us) server_negotiation.cc:407] Connection header received
0620 21:55:08.716543 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.716547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.716599 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.716658 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.717255 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41742: BlockingRecv error: recv got EOF from 127.0.0.1:41742 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 21:55:08.718225 23576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:08.716457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.0.193.253:40067)
0620 21:55:08.716586 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.716591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.717211 (+ 620us) server_negotiation.cc:407] Connection header received
0620 21:55:08.717389 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.717395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.717448 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.717502 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.718098 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41832: BlockingRecv error: recv got EOF from 127.0.0.1:41832 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 21:55:08.721232 23576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:08.720235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41848 (local address 127.0.193.253:40067)
0620 21:55:08.720444 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.720448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.720459 (+ 11us) server_negotiation.cc:407] Connection header received
0620 21:55:08.720531 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.720533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.720591 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.720685 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.721100 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41848: BlockingRecv error: recv got EOF from 127.0.0.1:41848 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 21:55:08.725684 23576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:08.724704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41858 (local address 127.0.193.253:40067)
0620 21:55:08.724825 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.724829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.724850 (+ 21us) server_negotiation.cc:407] Connection header received
0620 21:55:08.724897 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.724899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.724948 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.725020 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.725566 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41858: BlockingRecv error: recv got EOF from 127.0.0.1:41858 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 21:55:08.731060 23576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:08.729881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.0.193.253:40067)
0620 21:55:08.730067 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.730071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.730133 (+ 62us) server_negotiation.cc:407] Connection header received
0620 21:55:08.730288 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.730290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.730344 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.730434 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.730932 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41874: BlockingRecv error: recv got EOF from 127.0.0.1:41874 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260620 21:55:08.738994 23576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:08.737680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41878 (local address 127.0.193.253:40067)
0620 21:55:08.737913 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.737918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.737938 (+ 20us) server_negotiation.cc:407] Connection header received
0620 21:55:08.738123 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.738126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.738180 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.738268 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.738860 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41878: BlockingRecv error: recv got EOF from 127.0.0.1:41878 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260620 21:55:08.750646 23576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:08.749307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.0.193.253:40067)
0620 21:55:08.749468 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.749473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.749683 (+ 210us) server_negotiation.cc:407] Connection header received
0620 21:55:08.749795 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.749799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.749859 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.749964 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.750519 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41888: BlockingRecv error: recv got EOF from 127.0.0.1:41888 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 21:55:08.770272 23576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:08.769066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.0.193.253:40067)
0620 21:55:08.769224 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.769229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.769350 (+ 121us) server_negotiation.cc:407] Connection header received
0620 21:55:08.769476 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.769479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.769546 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.769651 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.770149 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41890: BlockingRecv error: recv got EOF from 127.0.0.1:41890 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 21:55:08.805848 23576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:08.804457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.0.193.253:40067)
0620 21:55:08.804597 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.804601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.804823 (+ 222us) server_negotiation.cc:407] Connection header received
0620 21:55:08.804889 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.804891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.805020 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.805120 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.805668 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41906: BlockingRecv error: recv got EOF from 127.0.0.1:41906 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 21:55:08.873597 23576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:08.872214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.0.193.253:40067)
0620 21:55:08.872422 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:08.872427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:08.872598 (+ 171us) server_negotiation.cc:407] Connection header received
0620 21:55:08.872775 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:08.872779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:08.872838 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:08.872937 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:08.873447 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41908: BlockingRecv error: recv got EOF from 127.0.0.1:41908 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260620 21:55:09.005589 23576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:09.004301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.0.193.253:40067)
0620 21:55:09.004459 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:09.004463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:09.004588 (+ 125us) server_negotiation.cc:407] Connection header received
0620 21:55:09.004707 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:09.004710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:09.004799 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:09.004891 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:09.005464 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41914: BlockingRecv error: recv got EOF from 127.0.0.1:41914 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 21:55:09.265375 23576 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 21:55:09.264144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.0.193.253:40067)
0620 21:55:09.264297 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:09.264301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:09.264387 (+ 86us) server_negotiation.cc:407] Connection header received
0620 21:55:09.264582 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:09.264585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:09.264638 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:09.264724 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 21:55:09.265201 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41928: BlockingRecv error: recv got EOF from 127.0.0.1:41928 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 21:55:09.432893 23665 catalog_manager.cc:1259] Loaded cluster ID: bc3d2fe147ee42cabace4f7c5e9e55b1
I20260620 21:55:09.432993 23665 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96: loading cluster ID for follower catalog manager: success
I20260620 21:55:09.434628 23665 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96: acquiring CA information for follower catalog manager: success
I20260620 21:55:09.435829 23665 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 821726f71c91403981d2eee34a1f6c96: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:55:09.461701 23679 catalog_manager.cc:1259] Loaded cluster ID: bc3d2fe147ee42cabace4f7c5e9e55b1
I20260620 21:55:09.461846 23679 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3: loading cluster ID for follower catalog manager: success
I20260620 21:55:09.463699 23679 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3: acquiring CA information for follower catalog manager: success
I20260620 21:55:09.464869 23679 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a091053f9044a1fa0f6c22e7ddcf5d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 21:55:09.783783 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:09.779801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.0.193.253:40067)
0620 21:55:09.780091 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:09.780096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:09.780123 (+ 27us) server_negotiation.cc:407] Connection header received
0620 21:55:09.780188 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:09.780191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:09.780361 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:09.780461 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:09.780706 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:09.781248 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:09.782170 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:09.782463 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:09.782562 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:09.782873 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:09.783412 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:09.783511 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:09.783597 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 21:55:09.783654 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260620 21:55:09.792276 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:09.787591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.0.193.252:37415)
0620 21:55:09.787950 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:09.787955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:09.787970 (+ 15us) server_negotiation.cc:407] Connection header received
0620 21:55:09.788180 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:09.788183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:09.788344 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:09.788507 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:09.788714 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:09.789427 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:09.790338 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:09.790591 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:09.790685 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:09.791130 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:09.791683 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:09.791791 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:09.792002 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 21:55:09.792086 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":169,"threads_started":1}
I20260620 21:55:09.800398 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:09.796453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.0.193.252:37415)
0620 21:55:09.796585 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:09.796590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:09.796733 (+ 143us) server_negotiation.cc:407] Connection header received
0620 21:55:09.796947 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:09.796954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:09.797021 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:09.797121 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:09.797363 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:09.798152 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:09.799176 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:09.799446 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:09.799563 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:09.799823 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:09.799915 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:09.799995 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:09.800133 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 21:55:09.800197 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 21:55:09.801024 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:09.796710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.0.193.253:40067)
0620 21:55:09.796876 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:09.796881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:09.796901 (+ 20us) server_negotiation.cc:407] Connection header received
0620 21:55:09.797038 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:09.797040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:09.797092 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:09.797175 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:09.797586 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:09.798337 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:09.799637 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:09.799989 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:09.800067 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:09.800469 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:09.800557 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:09.800632 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:09.800831 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 21:55:09.800902 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 21:55:09.802230 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 21:55:09.796260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57640 (local address 127.0.193.254:36683)
0620 21:55:09.796540 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 21:55:09.796547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 21:55:09.796565 (+ 18us) server_negotiation.cc:407] Connection header received
0620 21:55:09.796674 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 21:55:09.796681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 21:55:09.797103 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 21:55:09.797219 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 21:55:09.797614 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:09.798179 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:09.799635 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 21:55:09.799983 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 21:55:09.800085 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 21:55:09.800430 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 21:55:09.801118 (+ 688us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 21:55:09.801216 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 21:55:09.801987 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 21:55:09.802082 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20260620 21:55:09.806211 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 23428
I20260620 21:55:09.812929 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 23503
I20260620 21:55:09.819005 775 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBo3tyv/build/debug/bin/kudu with pid 23578
2026-06-20T21:55:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3323 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6541 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (298973 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST